2026-11-02T17:38:50.770724Z info FLAG: --concurrency="0" 2026-11-02T17:38:50.770742Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T17:38:50.770746Z info FLAG: --help="false" 2026-11-02T17:38:50.770748Z info FLAG: --log_as_json="false" 2026-11-02T17:38:50.770750Z info FLAG: --log_caller="" 2026-11-02T17:38:50.770752Z info FLAG: --log_output_level="default:info" 2026-11-02T17:38:50.770754Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T17:38:50.770758Z info FLAG: --log_target="[stdout]" 2026-11-02T17:38:50.770761Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T17:38:50.770763Z info FLAG: --outlierLogPath="" 2026-11-02T17:38:50.770765Z info FLAG: --profiling="true" 2026-11-02T17:38:50.770767Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T17:38:50.770769Z info FLAG: --proxyLogLevel="warning" 2026-11-02T17:38:50.770771Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T17:38:50.770773Z info FLAG: --stsPort="0" 2026-11-02T17:38:50.770775Z info FLAG: --templateFile="" 2026-11-02T17:38:50.770776Z info FLAG: --tokenManagerPlugin="" 2026-11-02T17:38:50.770783Z info FLAG: --vklog="0" 2026-11-02T17:38:50.770786Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T17:38:50.770914Z info Proxy role ips=[10.134.0.44] type=router id=maas-default-gateway-openshift-default-5db77fbd9f-nxpnm.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T17:38:50.770953Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T17:38:50.772041Z info cpu limit detected as 2, setting concurrency 2026-11-02T17:38:50.772237Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T17:38:50.772248Z info JWT policy is third-party-jwt 2026-11-02T17:38:50.772251Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T17:38:50.773737Z info Starting default Istio SDS Server 2026-11-02T17:38:50.773778Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T17:38:50.773797Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T17:38:50.773868Z info Opening status port 15020 2026-11-02T17:38:50.774438Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T17:38:50.777087Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T17:38:50.777935Z info Starting proxy agent 2026-11-02T17:38:50.777964Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T17:38:50.786727Z info sds Starting SDS grpc server 2026-11-02T17:38:50.786739Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T17:38:50.840529Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T17:38:50.945638Z info cache generated new workload certificate resourceName=default latency=169.392017ms ttl=23h59m59.054367234s 2026-11-02T17:38:50.945673Z info cache Root cert has changed, start rotating root cert 2026-11-02T17:38:50.945705Z info cache returned workload trust anchor from cache ttl=23h59m59.054295948s 2026-11-02T17:38:51.003442Z info ads ADS: new connection for node:1 2026-11-02T17:38:51.003526Z info cache returned workload trust anchor from cache ttl=23h59m58.996477423s 2026-11-02T17:38:51.003665Z info ads ADS: new connection for node:2 2026-11-02T17:38:51.003720Z info cache returned workload certificate from cache ttl=23h59m58.996282838s 2026-11-02T17:38:51.085999Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:39:00.074089Z info Readiness succeeded in 9.3058954s 2026-11-02T17:39:00.074485Z info Envoy proxy is ready [2026-11-02T17:39:45.797Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 113 324 194 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "c25f9952-77d2-47cc-bc22-66922d1afeda" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:44904 10.134.0.44:443 10.132.0.21:45936 - v1-chat-completions-path [2026-11-02T17:40:06.969Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 135 4624 74 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "8f784308-49ee-438c-8653-f8309a19c4f0" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:50850 10.134.0.44:443 10.132.0.21:53198 - v1-chat-completions-path 2026-11-02T17:46:44.863167Z info ads ADS: new connection for node:3 2026-11-02T17:46:44.868891Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T17:46:44.868941Z info cache read certificate from file resource=file-root:system 2026-11-02T17:52:10.913955Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:52:30.753719Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:52:34.941307Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:52:34.941711Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:52:34.810Z] "POST /anything/e2e-rt-sw-7a30a4-client HTTP/1.1" 200 - via_upstream - "-" 124 1764 131 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "0296dbad-87e1-47cb-a590-7d654ac93136" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:40368 10.134.0.44:443 10.132.0.21:41920 - llm.external-model-e2e-rt-sw-7a30a4-b69d508f4f72.2 2026-11-02T17:52:35.674762Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:52:35.675153Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:52:35.540Z] "POST /anything/e2e-rt-sw-7a30a4-client HTTP/1.1" 200 - via_upstream - "-" 124 1765 134 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "f2ed85a3-a31d-40f6-a21e-d9d614b4def3" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:40374 10.134.0.44:443 10.132.0.21:41928 - llm.external-model-e2e-rt-sw-7a30a4-b69d508f4f72.2 2026-11-02T17:52:46.757690Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:52:46.757987Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:52:46.615Z] "POST /anything/e2e-rt-sw-7a30a4-client HTTP/1.1" 200 - via_upstream - "-" 124 1757 142 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "c4f3947d-259e-4a5c-8a2a-bbcc48a2345c" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:37726 10.134.0.44:443 10.132.0.21:42908 - llm.external-model-e2e-rt-sw-7a30a4-b69d508f4f72.0 2026-11-02T17:52:48.387148Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:52:59.046986Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:52:59.047294Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:52:58.758Z] "POST /anything/e2e-rt-sw-7a30a4-client HTTP/1.1" 200 - via_upstream - "-" 124 1766 288 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "02694534-2dd6-457d-b429-ae3fba7932be" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:40368 10.134.0.44:443 10.132.0.21:60496 - llm.external-model-e2e-rt-sw-7a30a4-b69d508f4f72.0 2026-11-02T17:53:05.094652Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:53:05.094896Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T17:53:05.027Z] "POST /llm/e2e-rt-sw-7a30a4/anything/e2e-rt-sw-7a30a4-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 67 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "c02b3d2e-b133-41b4-9cc3-a3e74cabcae1" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-7a30a4.llm.svc.cluster.local - 10.134.0.44:443 10.132.0.21:54036 - llm.external-model-e2e-rt-sw-7a30a4-b69d508f4f72.2 2026-11-02T17:53:10.355476Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:53:10.355760Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:53:10.295Z] "POST /llm/e2e-rt-sw-7a30a4/anything/e2e-rt-sw-7a30a4-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 60 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "721b8bf1-8d49-4ae3-8d63-2e08380cdef8" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-7a30a4.llm.svc.cluster.local - 10.134.0.44:443 10.132.0.21:54048 - llm.external-model-e2e-rt-sw-7a30a4-b69d508f4f72.2 2026-11-02T17:53:12.600943Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:53:28.283861Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:53:28.284103Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T17:53:28.147Z] "POST /anything/e2e-rt-sw-7a30a4-client HTTP/1.1" 200 - via_upstream - "-" 124 1766 136 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "2d176e89-612f-4358-8bad-23f3e5f4dc7b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:38090 10.134.0.44:443 10.132.0.21:48998 - llm.external-model-e2e-rt-sw-7a30a4-b69d508f4f72.0 2026-11-02T17:53:46.703238Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:54:05.377532Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:54:11.240966Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:54:11.241198Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T17:54:11.110Z] "POST /anything/e2e-rt-eq-ca2488-client HTTP/1.1" 200 - via_upstream - "-" 124 1762 130 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "eb78b651-4d54-4f45-9b29-a5258a722cbf" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:43418 10.134.0.44:443 10.132.0.21:45042 - llm.external-model-e2e-rt-eq-ca2488-d8a0e9711142.0 2026-11-02T17:54:12.298747Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:54:12.299112Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 2026-11-02T17:54:12.916143Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-02T17:54:12.916938Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:54:12.917160Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:54:12.152Z] "POST /anything/e2e-rt-eq-ca2488-client HTTP/1.1" 200 - via_upstream - "-" 124 1763 147 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "f803150c-1982-4f0e-bec7-fc1ab772f313" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:55590 10.134.0.44:443 10.132.0.21:45046 - llm.external-model-e2e-rt-eq-ca2488-d8a0e9711142.0 [2026-11-02T17:54:12.798Z] "POST /anything/e2e-rt-eq-ca2488-client HTTP/1.1" 200 - via_upstream - "-" 124 1766 118 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "eda56904-0b0e-42fe-9613-6155366a3abb" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:40368 10.134.0.44:443 10.132.0.21:45054 - llm.external-model-e2e-rt-eq-ca2488-d8a0e9711142.2 2026-11-02T17:54:14.240474Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:54:14.240736Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T17:54:14.127Z] "POST /anything/e2e-rt-eq-ca2488-client HTTP/1.1" 200 - via_upstream - "-" 124 1765 113 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "293b088c-47dc-4ff7-a73d-1cea75edc6ab" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:43418 10.134.0.44:443 10.132.0.21:49716 - llm.external-model-e2e-rt-eq-ca2488-d8a0e9711142.2 2026-11-02T17:54:15.210031Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-02T17:54:15.210272Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:54:15.210511Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 2026-11-02T17:54:15.836776Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:54:15.836998Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:54:15.097Z] "POST /anything/e2e-rt-eq-ca2488-client HTTP/1.1" 200 - via_upstream - "-" 124 1764 113 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "1a47aac4-0b99-4a17-b5c0-5285a471205e" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:43418 10.134.0.44:443 10.132.0.21:49720 - llm.external-model-e2e-rt-eq-ca2488-d8a0e9711142.0 [2026-11-02T17:54:15.721Z] "POST /anything/e2e-rt-eq-ca2488-client HTTP/1.1" 200 - via_upstream - "-" 124 1764 115 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "95245977-e566-4443-a0eb-37fe9f38413b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:40374 10.134.0.44:443 10.132.0.21:49748 - llm.external-model-e2e-rt-eq-ca2488-d8a0e9711142.0 2026-11-02T17:54:39.267449Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:54:50.584142Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:55:01.969154Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:55:08.139180Z info ads ADS: "" 3 terminated 2026-11-02T17:55:23.656431Z info ads ADS: new connection for node:4 2026-11-02T17:55:23.692122Z info cache read certificate from file resource=file-root:system 2026-11-02T17:56:52.130981Z info ads ADS: "" 4 terminated with stream closed 2026-11-02T17:56:54.746512Z info ads ADS: new connection for node:5 2026-11-02T17:56:54.759281Z info cache read certificate from file resource=file-root:system 2026-11-02T17:58:01.673858Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:58:13.170101Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:58:29.021356Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:00:18.443747Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:01:08.999738Z info ads ADS: "" 5 terminated with stream closed 2026-11-02T18:01:09.132657Z info ads ADS: new connection for node:6 2026-11-02T18:01:09.139368Z info cache read certificate from file resource=file-root:system 2026-11-02T18:01:10.050336Z info ads ADS: "" 6 terminated 2026-11-02T18:01:26.571129Z info ads ADS: new connection for node:7 2026-11-02T18:01:26.576376Z info cache read certificate from file resource=file-root:system 2026-11-02T18:01:31.482991Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:02:02.640346Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:02:27.641402Z info ads ADS: "" 7 terminated with stream closed 2026-11-02T18:02:29.419173Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:02:30.801659Z info ads ADS: new connection for node:8 2026-11-02T18:02:30.806226Z info cache read certificate from file resource=file-root:system 2026-11-02T18:02:33.728502Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:02:35.987715Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:02:39.803785Z info ads ADS: "" 8 terminated with stream closed 2026-11-02T18:02:42.009639Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:02:43.409918Z info ads ADS: new connection for node:9 2026-11-02T18:02:43.414953Z info cache read certificate from file resource=file-root:system 2026-11-02T18:02:46.432093Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:02:49.639280Z info ads ADS: "" 9 terminated 2026-11-02T18:02:52.629716Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:02:52.663711Z info ads ADS: new connection for node:10 2026-11-02T18:02:52.677445Z info cache read certificate from file resource=file-root:system 2026-11-02T18:02:57.046332Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:02:58.981375Z info ads ADS: "" 10 terminated with stream closed 2026-11-02T18:03:00.919148Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:03:03.867522Z info ads ADS: new connection for node:11 2026-11-02T18:03:03.873122Z info cache read certificate from file resource=file-root:system 2026-11-02T18:03:05.643439Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:04:58.184324Z info ads ADS: "" 11 terminated 2026-11-02T18:05:07.611130Z info ads ADS: new connection for node:12 2026-11-02T18:05:07.622237Z info cache read certificate from file resource=file-root:system 2026-11-02T18:07:28.521228Z info ads ADS: new connection for node:13 2026-11-02T18:07:28.521438Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:07:28.521523Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:08:41.037163Z info ads ADS: "" 13 terminated 2026-11-02T18:09:37.860573Z info ads ADS: "" 12 terminated 2026-11-02T18:11:17.030777Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-02T18:11:44.894037Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T18:11:44.718Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 30 314 175 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "3aae00d1-52c0-4b2e-ab42-7efe5b50e71c" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:51154 10.134.0.44:443 10.132.0.21:59630 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T18:11:50.014500Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T18:11:49.903Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 111 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "59f290cb-7d21-4829-b73a-f7b097f3d48c" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:51154 10.134.0.44:443 10.132.0.21:59648 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T18:11:53.384626Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:12:15.242386Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:12:52.779176Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:13:12.336457Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:13:17.253182Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:13:24.006669Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-02T18:13:23.931Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 91 3616 75 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "c1afbf25-9536-45f0-8c59-69af27b30ea0" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.51:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:36402 10.134.0.44:443 10.132.0.21:56384 - v1-catch-all-model-routing 2026-11-02T18:13:29.644797Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:13:34.951083Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-02T18:13:34.884Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 66 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "f9ac82ee-7120-4c9f-9026-3158f1f3691f" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.51:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:36402 10.134.0.44:443 10.132.0.21:51244 - v1-catch-all-path 2026-11-02T18:13:48.954022Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:13:53.660844Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:14:08.215007Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:14:19.905243Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:14:23.962706Z info ads ADS: new connection for node:14 2026-11-02T18:14:23.962857Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:14:27.169710Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:15:11.272373Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:15:15.897Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 327 107 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "fb3eba74-2654-4eae-a284-c502cf3b870f" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.133.0.132:8000" outbound|54321||e2e-prio-pool-9e11459c-inference-pool-ip-d7a2bf95.llm.svc.cluster.local 10.134.0.44:41822 10.134.0.44:443 10.132.0.21:51594 - v1-model-routing 2026-11-02T18:15:36.722850Z info ads ADS: "" 14 terminated with stream closed 2026-11-02T18:15:37.059120Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:15:42.020998Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:15:59.087878Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:18:21.493118Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:18:49.554489Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:19:20.723090Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:19:34.763012Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:19:48.232761Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:19:59.727979Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:20:05.969402Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:20:10.755Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 80 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "0cf36d21-b26d-4eeb-8c97-adf1e710e79d" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:55164 10.134.0.44:443 10.132.0.21:46962 - v1-completions-path 2026-11-02T18:20:25.881262Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:24:51.325Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 71 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "64cbef6f-1de2-44c5-94c2-1153bbbd638a" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:50850 10.134.0.44:443 10.132.0.21:32912 - v1-completions-path 2026-11-02T18:25:22.692577Z info ads ADS: new connection for node:15 2026-11-02T18:25:22.699518Z info cache read certificate from file resource=file-root:system 2026-11-02T18:25:26.586814Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:25:27.784413Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:25:44.031389Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T18:25:44.031762Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T18:25:43.906Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1716 125 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "981096ad-a0eb-4294-b86f-e5cfaa1f1c49" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:44268 10.134.0.44:443 10.132.0.21:40570 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:25:51.027144Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:25:51.027375Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T18:25:50.863Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1717 163 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "31cec710-5c6e-4708-893d-14a4f9c79f5e" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:44278 10.134.0.44:443 10.132.0.21:54606 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:26:10.480218Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:26:10.480442Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T18:26:10.368Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1715 111 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "6306b1a2-55c0-4f48-9d2b-82808c864815" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:44278 10.134.0.44:443 10.132.0.21:59512 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:26:12.814837Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T18:26:12.815074Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T18:26:12.683Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1717 131 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "8e18bb85-82d5-4bfd-9bb0-01348f58996e" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:44268 10.134.0.44:443 10.132.0.21:59526 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:26:39.247899Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:26:39.248127Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T18:26:39.131Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1715 116 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "dc919827-d8ea-4b25-bcad-30224c109f46" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:44278 10.134.0.44:443 10.132.0.21:58666 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:26:43.940288Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T18:26:43.940558Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T18:26:43.829Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1716 110 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "26662ba8-ffa3-9e4c-a678-96a29615c014" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.44:44268 10.134.0.44:443 10.132.0.21:58690 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:27:15.356770Z info ads ADS: "" 15 terminated 2026-11-02T18:27:15.397901Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:27:18.918613Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:27:22.239414Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:27:22.239441Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:27:22.239444Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:27:22.239448Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T18:27:18.934Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 3304 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "32be9797-c583-4720-97d4-b07b2f640303" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.44:443 10.132.0.21:48216 - v1-completions-path 2026-11-02T18:27:25.445831Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-11-02T18:27:30.248378Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:27:50.019522Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:27:54.630118Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:28:00.116Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 299 70 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "86beac72-64df-4db2-97b8-abc4469f5cdc" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:55164 10.134.0.44:443 10.132.0.21:57798 - v1-completions-path 2026-11-02T18:28:02.043958Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T18:28:01.986Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 170 57 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "852162b9-ad1c-48ae-a857-24305f8bee88" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:55164 10.134.0.44:443 10.132.0.21:57802 - v1-catch-all-path 2026-11-02T18:28:15.540295Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:28:32.479508Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:28:41.020Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 72 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "8bd78059-a7aa-47df-b366-1ba790f2cc33" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:34362 10.134.0.44:443 10.132.0.21:34512 - v1-completions-path [2026-11-02T18:28:45.099Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 58 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "652d5e2b-24a7-4d77-8a98-ca97fbc7ebee" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:34362 10.134.0.44:443 10.132.0.21:49054 - v1-completions-path [2026-11-02T18:28:45.603Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 56 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "682db9ca-3488-4c85-b2dc-b93388874adc" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:34362 10.134.0.44:443 10.132.0.21:49064 - v1-completions-path 2026-11-02T18:29:03.560498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:29:14.525806Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:29:19.192789Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:29:48.896950Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:29:53.525406Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:30:07.408327Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:30:17.342351Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:30:27.182216Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:30:36.039383Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:30:44.205363Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:30:47.769Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 287 72 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "7f506b96-1c48-4989-90d7-c73ed7ae53f8" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:50850 10.134.0.44:443 10.132.0.21:34586 - v1-completions-path 2026-11-02T18:31:03.178711Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:31:09.599947Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:31:31.331822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:31:36.863004Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:31:40.487Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 0 44 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "cfcec290-3f2b-41ec-a32e-8a3594b49171" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.44:443 10.132.0.21:58896 - v1-completions-path 2026-11-02T18:31:48.150827Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:32:14.857865Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:32:25.952475Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:32:46.352980Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:32:54.402481Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:35:50.174215Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:18.072087Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:36.726118Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:41.735853Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:42.864452Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:37:42.864478Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:37:42.864481Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:37:42.864485Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T18:37:42.801Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 62 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "017b6766-7f88-48ff-aaaf-a420e38b640a" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.44:443 10.132.0.21:59620 - v1-completions-path 2026-11-02T18:37:55.566443Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:38:07.029670Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:38:16.786477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:38:48.988100Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-02T18:45:29.691397Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:45:34.319259Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:21.705116Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:51:24.466497Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:51:29.621465Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:51:50.685838Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:52:01.328908Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:52:07.940752Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:52:21.258Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 66 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "d2103aa1-8322-4575-90f1-ee4140bfe4d0" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:34362 10.134.0.44:443 10.132.0.21:59860 - v1-completions-path 2026-11-02T18:52:36.872465Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:52:43.161366Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:52:49.344566Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:52:59.076Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 65 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "af9a12ef-fc6f-4157-af27-d9a723e8736b" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:55164 10.134.0.44:443 10.132.0.21:50868 - v1-completions-path [2026-11-02T18:53:00.874Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 57 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "2c995622-f8f9-4d08-ab81-ec188e2e1c51" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:34362 10.134.0.44:443 10.132.0.21:50876 - v1-completions-path [2026-11-02T18:53:02.685Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 60 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "7b52b3c1-2f62-44f0-a90f-7e5ab2057d78" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:34362 10.134.0.44:443 10.132.0.21:50888 - v1-completions-path 2026-11-02T18:53:06.235958Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T18:53:06.122Z] "DELETE /v1/api-keys/9915ea99-b1f4-4bef-aea8-67deeacc1fb1 HTTP/1.1" 200 - via_upstream - "-" 0 365 113 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "f47718c6-745a-43dc-b3a0-7ad2bc712ac5" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "10.134.0.123:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:57380 10.134.0.44:443 10.132.0.21:55720 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T18:53:19.795750Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:30.723732Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:37.543361Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:43.730295Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:55.144801Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:18.657846Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:29.443275Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:34.319878Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:52.752694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:57.330236Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:19.457574Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:39.475978Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:50.479523Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:01.549485Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:12.504075Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:23.624050Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:34.424130Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:45.301100Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:56.117118Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:06.833819Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:17.602728Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:28.179927Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T18:57:28.179951Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T18:57:28.179955Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T18:57:28.179958Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-02T18:57:28.154Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 25 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "74e356ec-94f9-4c9e-9042-73d029bad1e5" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.44:443 10.132.0.21:52856 - v1-completions-path 2026-11-02T18:57:31.582753Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:57:31.582777Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:57:31.582780Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:57:31.582784Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T18:57:31.557Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 24 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "76a75cf5-a0f3-44fe-8a67-a32a39b66565" "maas.apps.02a4e30a9ae600e3eac2.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.44:443 10.132.0.21:52888 - v1-completions-path