2026-10-31T23:17:33.842126Z info FLAG: --concurrency="0" 2026-10-31T23:17:33.842144Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-31T23:17:33.842148Z info FLAG: --help="false" 2026-10-31T23:17:33.842150Z info FLAG: --log_as_json="false" 2026-10-31T23:17:33.842152Z info FLAG: --log_caller="" 2026-10-31T23:17:33.842154Z info FLAG: --log_output_level="default:info" 2026-10-31T23:17:33.842156Z info FLAG: --log_stacktrace_level="default:none" 2026-10-31T23:17:33.842160Z info FLAG: --log_target="[stdout]" 2026-10-31T23:17:33.842163Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-31T23:17:33.842165Z info FLAG: --outlierLogPath="" 2026-10-31T23:17:33.842167Z info FLAG: --profiling="true" 2026-10-31T23:17:33.842169Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-31T23:17:33.842171Z info FLAG: --proxyLogLevel="warning" 2026-10-31T23:17:33.842173Z info FLAG: --serviceCluster="istio-proxy" 2026-10-31T23:17:33.842175Z info FLAG: --stsPort="0" 2026-10-31T23:17:33.842177Z info FLAG: --templateFile="" 2026-10-31T23:17:33.842179Z info FLAG: --tokenManagerPlugin="" 2026-10-31T23:17:33.842181Z info FLAG: --vklog="0" 2026-10-31T23:17:33.842184Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-31T23:17:33.842409Z info Proxy role ips=[10.133.0.37] type=router id=maas-default-gateway-openshift-default-6cc458d887-dh8ng.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-31T23:17:33.842457Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-31T23:17:33.843510Z info cpu limit detected as 2, setting concurrency 2026-10-31T23:17:33.843715Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-31T23:17:33.843725Z info JWT policy is third-party-jwt 2026-10-31T23:17:33.843727Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-31T23:17:33.844876Z info Starting default Istio SDS Server 2026-10-31T23:17:33.844900Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-31T23:17:33.844916Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-31T23:17:33.845293Z info Opening status port 15020 2026-10-31T23:17:33.846648Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-31T23:17:33.847829Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-31T23:17:33.848553Z info Starting proxy agent 2026-10-31T23:17:33.848582Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-31T23:17:33.857992Z info sds Starting SDS grpc server 2026-10-31T23:17:33.858008Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-31T23:17:33.907508Z info cache generated new workload certificate resourceName=default latency=60.637225ms ttl=23h59m59.092494236s 2026-10-31T23:17:33.907536Z info cache Root cert has changed, start rotating root cert 2026-10-31T23:17:33.907562Z info cache returned workload trust anchor from cache ttl=23h59m59.09243814s 2026-10-31T23:17:33.910767Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-31T23:17:34.120527Z info ads ADS: new connection for node:1 2026-10-31T23:17:34.120592Z info cache returned workload certificate from cache ttl=23h59m58.879410255s 2026-10-31T23:17:34.120607Z info ads ADS: new connection for node:2 2026-10-31T23:17:34.120685Z info cache returned workload trust anchor from cache ttl=23h59m58.879316585s 2026-10-31T23:17:34.198508Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:17:41.864646Z info Readiness succeeded in 8.025478571s 2026-10-31T23:17:41.865072Z info Envoy proxy is ready [2026-10-31T23:18:45.681Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 389 79 - "52.71.36.72,10.132.0.9" "curl/7.76.1" "d5d3663f-cb5c-4f38-9355-ae6d1f1bc480" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:44452 10.133.0.37:443 10.132.0.9:41718 - v1-model-routing [2026-10-31T23:18:46.100Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 55 - "52.71.36.72,10.132.0.9" "curl/7.76.1" "2bedaf74-e482-4738-bc7f-1340fa42cfa2" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.132.0.9:41732 - v1-model-routing 2026-10-31T23:18:47.020821Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:18:47.020857Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:18:47.020871Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T23:18:47.020875Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-31T23:18:46.946Z] "GET /v1/subscriptions HTTP/1.1" 200 - via_upstream - "-" 0 488 74 - "52.71.36.72,10.132.0.9" "curl/7.76.1" "3865d829-9de2-4b66-b128-850a6cb403c1" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:36580 10.133.0.37:443 10.132.0.9:41736 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T23:22:54.384496Z info ads ADS: new connection for node:3 2026-10-31T23:22:54.395446Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-31T23:22:54.395516Z info cache read certificate from file resource=file-root:system 2026-10-31T23:22:56.255625Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:23:02.679053Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:23:25.998237Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:23:29.074307Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:23:29.074343Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:23:29.074357Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:23:29.074362Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-31T23:23:28.640Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 74 316 434 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "b810ae7a-4507-4ca9-aa1b-628abf57ca21" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:48706 10.133.0.37:443 10.132.0.9:44184 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T23:23:37.163245Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:23:56.811075Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:24:04.866182Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:24:04.866219Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:24:04.866233Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:24:04.866239Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T23:24:04.866451Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T23:24:04.750Z] "POST /anything/e2e-rt-sw-e72e54-client HTTP/1.1" 200 - via_upstream - "-" 124 1773 116 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "7920e7f3-c32b-420f-beb8-4217ba934826" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:56966 10.133.0.37:443 10.132.0.9:47974 - llm.external-model-e2e-rt-sw-e72e54-070c9f0e553a.0 2026-10-31T23:24:10.306906Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:24:10.306935Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:24:10.306945Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:24:10.306950Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T23:24:10.307096Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T23:24:10.191Z] "POST /anything/e2e-rt-sw-e72e54-client HTTP/1.1" 200 - via_upstream - "-" 124 1776 115 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "b89d945d-bf52-400b-b06a-e6af81ef300f" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:56966 10.133.0.37:443 10.132.0.9:47982 - llm.external-model-e2e-rt-sw-e72e54-070c9f0e553a.0 2026-10-31T23:24:18.932615Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:24:35.412689Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:24:40.289928Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:24:40.289955Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:24:40.289964Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:24:40.289968Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T23:24:40.290127Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T23:24:40.149Z] "POST /anything/e2e-rt-eq-176064-client HTTP/1.1" 200 - via_upstream - "-" 124 1774 140 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "9a06a524-765c-41bb-9418-722325ac9520" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:53636 10.133.0.37:443 10.132.0.9:38470 - llm.external-model-e2e-rt-eq-176064-3cea81747a79.2 2026-10-31T23:25:10.792534Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:25:15.347006Z info ads ADS: "" 3 terminated 2026-10-31T23:25:15.813050Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:25:27.425560Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:27:34.733923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:29:30.762900Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:30:14.158883Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:30:27.349780Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:30:27.349814Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:30:27.349828Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T23:30:27.349833Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T23:30:27.272Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 77 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "39d8c21b-70ba-4153-8179-42424c57d8d5" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:46204 10.133.0.37:443 10.132.0.9:59144 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T23:30:29.065422Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:30:29.065456Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:30:29.065471Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T23:30:29.065475Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T23:30:28.973Z] "DELETE /v1/api-keys/db3b166c-f341-42fa-9bb1-ee260393a365 HTTP/1.1" 200 - via_upstream - "-" 0 321 92 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "5607cd7d-3191-4526-b1d4-f4ea46a06808" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:46204 10.133.0.37:443 10.132.0.9:59170 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T23:31:09.663Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 78 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "94bc3dfc-42b6-42b1-b93e-cdf37adae09f" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:46342 10.133.0.37:443 10.132.0.9:33334 - v1-completions-path 2026-10-31T23:31:10.232968Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-31T23:31:10.937059Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:31:10.937096Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:31:10.937115Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:31:10.937121Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-31T23:31:09.859Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 375 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "014e6821-13ca-4725-9342-5361a8e0661d" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:46342 10.133.0.37:443 10.132.0.9:33342 - v1-completions-path [2026-10-31T23:31:10.927Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 9 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "873c9ebf-5e14-4016-a6db-cdd1dbf14e8e" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:32816 10.133.0.37:443 10.132.0.9:33352 - odh-ai-gateway-infra.maas-api-route.4 2026-10-31T23:31:11.201533Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:31:11.201578Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:31:11.201603Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:31:11.201610Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-31T23:31:11.118Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 83 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "415c3596-3d0e-4cc5-b78f-3d2d383d4841" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:32816 10.133.0.37:443 10.132.0.9:33368 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T23:39:31.857600Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-31T23:39:31.793Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 113 334 69 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "23ae1e2d-58d2-4a02-9028-90b54d37e0cb" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:44452 10.133.0.37:443 10.132.0.9:46478 - v1-chat-completions-path 2026-10-31T23:39:42.841239Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:39:42.841271Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T23:39:42.841285Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T23:39:42.841289Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-31T23:39:42.830Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 11 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "5fc7a5fa-e01a-4d13-9be6-8b505835244e" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:58908 10.133.0.37:443 10.132.0.9:46784 - odh-ai-gateway-infra.maas-api-route.4 [2026-10-31T23:48:04.585Z] "GET /v1/tenants HTTP/1.1" 404 NR route_not_found - "-" 0 0 10 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "0d683c0a-92c3-469f-93d3-931ca7f0db31" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "-" - - 10.133.0.37:443 10.132.0.9:54380 - - 2026-10-31T23:50:16.696336Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-31T23:51:57.963746Z info ads ADS: new connection for node:4 2026-10-31T23:51:57.968963Z info cache read certificate from file resource=file-root:system 2026-10-31T23:58:52.152525Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:59:25.440816Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T23:59:29.487405Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:59:29.487439Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:59:29.487454Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T23:59:29.487459Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T23:59:29.376Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 66 308 110 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "971ebf83-2c6e-495d-bc5f-798cfeaefa9d" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:56114 10.133.0.37:443 10.132.0.9:36860 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T00:00:03.970962Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:00:42.043865Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:01:35.416458Z info ads ADS: new connection for node:5 2026-11-01T00:01:35.416609Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T00:01:35.416634Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T00:01:38.476323Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:01:43.984969Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:01:43.985007Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:01:43.985022Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:01:43.985027Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-01T00:01:43.985237Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-01T00:01:43.857Z] "POST /anything/e2e-dft-e0da35-2c96b6 HTTP/1.1" 200 - via_upstream - "-" 109 1772 128 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "f48705bc-c86b-4bac-bd82-7003ab7bccf1" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:54542 10.133.0.37:443 10.132.0.9:56068 - llm.external-model-e2e-dft-e0da35-744579d2ac17.0 2026-11-01T00:01:57.754018Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:02:20.454481Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:06:06.483826Z info ads ADS: "" 5 terminated with stream closed 2026-11-01T00:06:08.426546Z info ads ADS: "" 4 terminated with stream closed 2026-11-01T00:06:42.056110Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-11-01T00:13:05.039124Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:14:15.844156Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:17:08.893462Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:19:28.731026Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:22:40.137663Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-01T00:28:26.657844Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:28:49.795310Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:29:23.998037Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:29:27.311Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 5518 5791 81 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "6f4d39c1-579d-4d0f-94c2-3045a13697dd" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.0.44:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:44744 10.133.0.37:443 10.132.0.9:58998 - v1-chat-completions-path [2026-11-01T00:29:27.615Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 144 409 61 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "33b8e7fb-41cf-406c-8112-dfc117ea1429" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.0.44:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:44744 10.133.0.37:443 10.132.0.9:59018 - v1-chat-completions-path 2026-11-01T00:29:39.785883Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:29:42.206399Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:29:48.591548Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:29:55.698100Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:30:05.588247Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:30:20.390399Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:30:31.426936Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:30:33.127118Z info ads ADS: new connection for node:6 2026-11-01T00:30:33.127384Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T00:30:33.127475Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T00:30:34.478313Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:31:15.914496Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:31:21.519Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 333 87 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "a066d379-460c-4f15-9114-558efa5fd43f" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.1.164:8000" outbound|54321||e2e-prio-pool-ca03fe61-inference-pool-ip-13923081.llm.svc.cluster.local 10.133.0.37:33194 10.133.0.37:443 10.132.0.9:39366 - v1-model-routing 2026-11-01T00:31:35.232927Z info ads ADS: "" 6 terminated 2026-11-01T00:31:35.280644Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:31:37.704429Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:31:40.533145Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:31:45.641258Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:34:00.684348Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:34:00.684381Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:34:00.684393Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:34:00.684398Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-01T00:34:00.618Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 66 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "1998ac81-90bf-41d0-a7c6-9a95da5da540" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:48242 10.133.0.37:443 10.132.0.9:35200 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T00:34:05.257209Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:34:21.831984Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:34:36.275881Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:34:36.275940Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:34:36.275969Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T00:34:36.275980Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-01T00:34:36.216Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 628 59 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "48a6aa9c-36c6-4313-96fb-4b2264cb968d" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:60018 10.133.0.37:443 10.132.0.9:36632 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T00:34:38.316141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:34:50.274124Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:34:53.837760Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:34:53.837800Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:34:53.837820Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:34:53.837826Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-01T00:34:53.759Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 84 326 78 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "794b6988-a65a-43a0-95af-afca0cc11de9" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:48242 10.133.0.37:443 10.132.0.9:37126 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T00:34:54.115088Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:35:05.872287Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:35:09.407968Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:35:12.361Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 83 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "f6249452-9e27-4e5a-86c4-4f375a84486b" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45428 10.133.0.37:443 10.132.0.9:46474 - v1-completions-path [2026-11-01T00:35:12.467Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 61 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "aef8dde4-d5a4-4192-bf91-35e43967dcc0" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45438 10.133.0.37:443 10.132.0.9:46478 - v1-completions-path 2026-11-01T00:35:25.105221Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:36:12.290715Z info ads ADS: new connection for node:7 2026-11-01T00:36:12.294100Z info cache read certificate from file resource=file-root:system 2026-11-01T00:36:13.402681Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:36:14.975997Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:36:34.737964Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:36:34.737993Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:36:34.738002Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:36:34.738006Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T00:36:34.738141Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-01T00:36:34.628Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1711 109 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "92346a29-3751-42a9-ae64-e43bd925ccee" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:37778 10.133.0.37:443 10.132.0.9:34964 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-01T00:37:11.875989Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:37:11.876018Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:37:11.876026Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:37:11.876031Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T00:37:11.876167Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-01T00:37:11.774Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1715 101 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "7f0158fd-4a77-4b17-8916-fbc944734c29" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:37778 10.133.0.37:443 10.132.0.9:49562 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-01T00:37:23.406525Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:37:23.406557Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:37:23.406568Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T00:37:23.406572Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-01T00:37:23.406744Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-01T00:37:23.299Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1716 107 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "ded3833b-ee2c-448d-82e4-1bcca532b3fe" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:55338 10.133.0.37:443 10.132.0.9:39978 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-01T00:37:56.784376Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:37:56.930759Z info ads ADS: "" 7 terminated 2026-11-01T00:37:58.299797Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:38:05.070056Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:38:20.699927Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:38:25.248136Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:38:40.516864Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:38:56.562784Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:39:03.092Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 64 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "cd41b335-0693-477e-a12f-f4a9456f7266" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45438 10.133.0.37:443 10.132.0.9:48650 - v1-completions-path [2026-11-01T00:39:03.177Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 57 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "94ef6388-7374-4e25-8f6d-e2535cccb1eb" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45438 10.133.0.37:443 10.132.0.9:48660 - v1-completions-path [2026-11-01T00:39:03.626Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 57 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "73b04411-7bfc-4497-996a-c20202bf5874" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45438 10.133.0.37:443 10.132.0.9:48670 - v1-completions-path [2026-11-01T00:39:04.008Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 58 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "b39e081d-175d-466a-9718-b973668481ff" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45428 10.133.0.37:443 10.132.0.9:48690 - v1-completions-path 2026-11-01T00:39:20.004783Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:39:30.298167Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:39:34.880296Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:39:39.768Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 57 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "727dca33-6bb3-447e-9915-edbf1f7da52e" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45428 10.133.0.37:443 10.132.0.9:53838 - v1-completions-path 2026-11-01T00:39:51.831339Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:39:54.175025Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:01.252356Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:13.183175Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:18.405790Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:26.731925Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:31.964311Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:40:38.304Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 295 66 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "5fa276ff-dcb3-4f4b-af35-ca9556720587" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:56892 10.133.0.37:443 10.132.0.9:33166 - v1-completions-path 2026-11-01T00:40:50.306278Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:53.247235Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:41:17.110452Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:41:22.772Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 61 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "6cd78eb6-3ab1-448b-865e-c5747ce991e2" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45438 10.133.0.37:443 10.132.0.9:57910 - v1-completions-path 2026-11-01T00:41:35.066400Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:41:45.346790Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:45:26.171844Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:45:28.775Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 73 288 98 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "dc19a140-9855-4bbb-9d9c-f4294fbe1347" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.45:8000" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45476 10.133.0.37:443 10.132.0.9:41358 - v1-completions-path 2026-11-01T00:45:41.551036Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:45:43.852485Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:45:50.973773Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:46:10.756368Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:46:14.358448Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:46:17.672Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 61 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "d617f731-2985-45a9-a227-ac113ab394b7" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45438 10.133.0.37:443 10.132.0.9:60182 - v1-completions-path [2026-11-01T00:46:18.326Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 56 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "a80bd85e-dbce-4097-8cec-a1019edb578d" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45438 10.133.0.37:443 10.132.0.9:60186 - v1-completions-path 2026-11-01T00:46:29.960238Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:46:40.353885Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:46:45.437444Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:46:50.008Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 62 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "84ec203a-e82d-4d7e-ae30-58e61bc75341" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45438 10.133.0.37:443 10.132.0.9:38736 - v1-completions-path 2026-11-01T00:47:02.441793Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:06.890075Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:10.421268Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:47:17.413Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 58 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "88481f46-838b-492e-bf04-fbcac25a8d52" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.132.0.9:40800 - v1-completions-path [2026-11-01T00:47:19.133Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 68 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "ae98db12-6a3f-41ed-8d89-04eca1e3fb0f" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45428 10.133.0.37:443 10.132.0.9:40810 - v1-completions-path 2026-11-01T00:47:33.124658Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:43.453266Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:47.906318Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:51.352654Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:56.284169Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:48:01.919Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 63 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "41b1cda0-a3bb-45da-87da-5ddcb52fa4f0" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:45428 10.133.0.37:443 10.132.0.9:50734 - v1-completions-path [2026-11-01T00:48:02.683Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 57 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "f0121c76-3b5e-468c-ad7a-d59662ef0218" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.132.0.9:50736 - v1-completions-path 2026-11-01T00:48:14.779782Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:25.102633Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:27.307364Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:36.907608Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:40.471043Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:58.330388Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:49:00.579481Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:49:19.029910Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:49:29.476374Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:49:40.006150Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:49:50.299967Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:00.625627Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:10.970053Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:21.358862Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:31.692106Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:41.992829Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:52.348498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:51:19.235100Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=4