2026-10-31T05:06:41.198111Z info FLAG: --concurrency="0" 2026-10-31T05:06:41.198142Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-31T05:06:41.198147Z info FLAG: --help="false" 2026-10-31T05:06:41.198149Z info FLAG: --log_as_json="false" 2026-10-31T05:06:41.198151Z info FLAG: --log_caller="" 2026-10-31T05:06:41.198153Z info FLAG: --log_output_level="default:info" 2026-10-31T05:06:41.198155Z info FLAG: --log_stacktrace_level="default:none" 2026-10-31T05:06:41.198160Z info FLAG: --log_target="[stdout]" 2026-10-31T05:06:41.198162Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-31T05:06:41.198165Z info FLAG: --outlierLogPath="" 2026-10-31T05:06:41.198168Z info FLAG: --profiling="true" 2026-10-31T05:06:41.198170Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-31T05:06:41.198172Z info FLAG: --proxyLogLevel="warning" 2026-10-31T05:06:41.198174Z info FLAG: --serviceCluster="istio-proxy" 2026-10-31T05:06:41.198176Z info FLAG: --stsPort="0" 2026-10-31T05:06:41.198178Z info FLAG: --templateFile="" 2026-10-31T05:06:41.198180Z info FLAG: --tokenManagerPlugin="" 2026-10-31T05:06:41.198183Z info FLAG: --vklog="0" 2026-10-31T05:06:41.198186Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-31T05:06:41.198342Z info Proxy role ips=[10.134.1.45] type=router id=maas-default-gateway-openshift-default-7d64fcddb7-t74c9.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-31T05:06:41.198393Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-31T05:06:41.199830Z info cpu limit detected as 2, setting concurrency 2026-10-31T05:06:41.200079Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-31T05:06:41.200092Z info JWT policy is third-party-jwt 2026-10-31T05:06:41.200095Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-31T05:06:41.201763Z info Starting default Istio SDS Server 2026-10-31T05:06:41.201794Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-31T05:06:41.201817Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-31T05:06:41.202322Z info Opening status port 15020 2026-10-31T05:06:41.203173Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-31T05:06:41.203846Z info sds Starting SDS grpc server 2026-10-31T05:06:41.203867Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-31T05:06:41.204816Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-31T05:06:41.205993Z info Starting proxy agent 2026-10-31T05:06:41.206036Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-31T05:06:41.306776Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-31T05:06:41.401701Z info cache generated new workload certificate resourceName=default latency=197.749715ms ttl=23h59m59.598305121s 2026-10-31T05:06:41.401762Z info cache Root cert has changed, start rotating root cert 2026-10-31T05:06:41.401806Z info cache returned workload trust anchor from cache ttl=23h59m59.598194409s 2026-10-31T05:06:41.488720Z info ads ADS: new connection for node:1 2026-10-31T05:06:41.488827Z info cache returned workload certificate from cache ttl=23h59m59.511175148s 2026-10-31T05:06:41.489171Z info ads ADS: new connection for node:2 2026-10-31T05:06:41.489248Z info cache returned workload trust anchor from cache ttl=23h59m59.510753146s 2026-10-31T05:06:41.586982Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:06:49.767453Z info Readiness succeeded in 8.572279337s 2026-10-31T05:06:49.767781Z info Envoy proxy is ready 2026-10-31T05:06:50.379809Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:07:07.288630Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:07:18.910545Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:07:22.484943Z info ads ADS: new connection for node:3 2026-10-31T05:07:22.485188Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-31T05:07:22.485224Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-31T05:07:25.821341Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:08:10.501881Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:08:16.059Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 333 104 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "e6365beb-73f3-4012-b47f-93b41526e2d6" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.133.0.214:8000" outbound|54321||e2e-prio-pool-36410873-inference-pool-ip-2851bf73.llm.svc.cluster.local 10.134.1.45:56764 10.134.1.45:443 10.132.0.19:57334 - v1-model-routing [2026-10-31T05:08:17.288Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 349 73 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "a3eb4ef1-9e46-4ae9-b4a1-79262bd0e92d" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.133.0.214:8000" outbound|54321||e2e-prio-pool-36410873-inference-pool-ip-2851bf73.llm.svc.cluster.local 10.134.1.45:56770 10.134.1.45:443 10.132.0.19:57340 - v1-model-routing 2026-10-31T05:08:35.332339Z info ads ADS: "" 3 terminated with stream closed 2026-10-31T05:08:35.379455Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:08:40.231473Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:08:40.231519Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:08:40.231534Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T05:08:40.231538Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T05:08:40.112Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 90 332 119 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "a274135d-f28c-4099-ac55-b1ccc8e73b62" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.45:57918 10.134.1.45:443 10.132.0.19:59458 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T05:08:41.856695Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:08:54.791865Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:11:12.983855Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T05:11:12.983906Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T05:11:12.983930Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T05:11:12.983938Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-31T05:11:12.837Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 33 325 146 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "2bc36b82-0f9c-48ce-aa55-2aeed2367a63" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.45:36744 10.134.1.45:443 10.132.0.19:44636 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T05:11:32.028942Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:11:59.763024Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:12:36.277164Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:12:51.391816Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:13:00.249314Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:13:13.319962Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:13:17.753645Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:13:36.090084Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:16:15.310Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 279 96 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "cbc5d5d0-2431-4490-924d-52d656cde94f" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:55728 10.134.1.45:443 10.132.0.19:46712 - v1-completions-path 2026-10-31T05:17:00.612282Z info ads ADS: new connection for node:4 2026-10-31T05:17:00.617511Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-31T05:17:00.617569Z info cache read certificate from file resource=file-root:system 2026-10-31T05:17:04.343804Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:17:05.732540Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:17:08.746626Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:17:08.746663Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:17:08.746678Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T05:17:08.746683Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T05:17:08.611Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 134 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "51df7af8-3a40-478b-9608-55d4c22b5ab9" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.45:34344 10.134.1.45:443 10.132.0.19:49978 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T05:17:18.279934Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:17:18.279974Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:17:18.279987Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:17:18.279992Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T05:17:18.280205Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T05:17:18.065Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1722 214 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "a9a2acfb-e839-4228-b1e8-d1bb309ed0ea" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.1.45:56060 10.134.1.45:443 10.132.0.19:49986 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-31T05:17:56.466574Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-10-31T05:17:56.545340Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:17:56.545374Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:17:56.545386Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T05:17:56.545391Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:17:56.545648Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T05:17:56.408Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 137 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "56cb5c7b-9b0f-4dfd-9237-3c1350caeda5" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.1.45:56060 10.134.1.45:443 10.132.0.19:36546 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-31T05:18:15.608481Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:18:15.608518Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:18:15.608531Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T05:18:15.608537Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:18:15.608741Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T05:18:15.468Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1719 140 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "e7096c75-8936-42aa-84b0-dadc5bbdbeec" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.1.45:56060 10.134.1.45:443 10.132.0.19:36128 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-31T05:18:35.632040Z info ads ADS: "" 4 terminated 2026-10-31T05:18:35.674693Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:18:36.260464Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-31T05:18:36.260487Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-31T05:18:36.260491Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-31T05:18:36.260495Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-31T05:18:36.222Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 37 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "21c80da0-32e1-4af8-9c45-3ee204900d2f" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.45:443 10.132.0.19:37344 - v1-completions-path 2026-10-31T05:18:38.705413Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:18:48.263689Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:18:52.449Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 78 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "d820dd9c-95bd-437a-834a-941beeb730cb" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:47146 10.134.1.45:443 10.132.0.19:43830 - v1-completions-path 2026-10-31T05:19:07.132967Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:19:11.841102Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:19:28.320033Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:19:45.232398Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:19:55.162566Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-10-31T05:19:55.087Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 83 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "69681b58-d713-4ca1-8766-a93d4c36f7d8" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:53360 10.134.1.45:443 10.132.0.19:57164 - v1-completions-path [2026-10-31T05:19:57.251Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 64 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "13bc7faf-b01e-4c3d-9148-37dce0d0b9f2" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:47146 10.134.1.45:443 10.132.0.19:57184 - v1-completions-path [2026-10-31T05:19:57.499Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "e41e46f9-78b8-435e-a184-87221bf8603a" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:47146 10.134.1.45:443 10.132.0.19:57198 - v1-completions-path [2026-10-31T05:19:58.563Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 59 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "117d65eb-17bd-43db-ad8e-a78e38eaf9d0" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:53360 10.134.1.45:443 10.132.0.19:60558 - v1-completions-path 2026-10-31T05:20:16.321067Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:20:27.062617Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:20:33.304703Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:20:48.462Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "6af41a8e-87bf-4cf4-bd08-5c0219fafea9" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:47146 10.134.1.45:443 10.132.0.19:38518 - v1-completions-path 2026-10-31T05:21:01.929032Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:21:06.616550Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:21:19.917966Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:21:26.189415Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:21:40.840896Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:21:45.287172Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:21:57.102789Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:21:57.041Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 279 102 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "fb10ba87-2e02-4a69-b67f-d84927531ab2" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:39140 10.134.1.45:443 10.132.0.19:38380 - v1-completions-path 2026-10-31T05:22:12.321342Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:22:17.658Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 66 18 63 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "1160fffc-99e1-4697-9351-38f2243b9bb1" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.45:443 10.132.0.19:54494 - v1-completions-path 2026-10-31T05:22:18.643311Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:22:38.642401Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:22:44.139289Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:22:46.797022Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:22:46.797070Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:22:46.797093Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:22:46.797101Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T05:22:46.678Z] "DELETE /v1/api-keys/de23a721-1179-4f73-b312-62f6b329c89a HTTP/1.1" 200 - via_upstream - "-" 0 375 118 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "55229b24-18ad-41e1-b091-f5d8ebfef766" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.45:53480 10.134.1.45:443 10.132.0.19:46488 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T05:22:55.269743Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:23:19.551275Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:23:30.309390Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:23:49.922645Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:23:57.482391Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:26:18.762Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 51 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "1950a5a8-3ab8-44e5-8444-68b5ccd05da3" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.45:443 10.132.0.19:46192 - v1-completions-path 2026-10-31T05:26:22.908580Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:27:30.058174Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:27:30.058216Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:27:30.058235Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T05:27:30.058239Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T05:27:29.931Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 127 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "04f18ad2-803c-4596-a3c2-2d79ef269d89" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.45:52622 10.134.1.45:443 10.132.0.19:44818 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T05:27:52.893057Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:27:55.577Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 429 - - "-" 73 18 99 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "b7cd5845-e1a2-4ff0-ab78-0bfedec46326" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.45:443 10.132.0.19:38652 - v1-completions-path 2026-10-31T05:28:12.591342Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:28:17.857460Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:28:18.586408Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T05:28:18.586455Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T05:28:18.586459Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T05:28:18.586463Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-31T05:28:18.533Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 53 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "c9ba8654-fc23-41e5-9c74-aff9cb18b80b" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.45:443 10.132.0.19:42932 - v1-completions-path 2026-10-31T05:28:31.548700Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:28:45.311537Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:28:50.689666Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:28:54.662Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 72 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "1986e499-f4a5-40f5-b4a7-7a0b581b57c3" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:53360 10.134.1.45:443 10.132.0.19:35472 - v1-completions-path 2026-10-31T05:29:10.386587Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:29:21.135284Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:29:27.680498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:29:57.080042Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:30:03.801394Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:30:10.126672Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:30:26.913Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 67 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "306eee98-b299-440c-9500-ae738d98fbc5" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:47146 10.134.1.45:443 10.132.0.19:40472 - v1-completions-path 2026-10-31T05:30:42.045336Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:30:52.548528Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:30:59.491323Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:31:05.847933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:31:13.781Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 67 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "82ef136e-f8c1-435b-a8cc-7b53c58d6997" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:47146 10.134.1.45:443 10.132.0.19:41286 - v1-completions-path 2026-10-31T05:31:17.925564Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:31:24.987Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 68 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "cafb7cd7-c162-4c60-a47e-8a795d93c9d9" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.45:47146 10.134.1.45:443 10.132.0.19:46012 - v1-completions-path 2026-10-31T05:31:40.817308Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:31:51.610055Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:31:56.828234Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:32:12.816367Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:32:19.226297Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:32:24.562196Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:32:48.283641Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:32:59.396586Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:33:19.276933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:33:30.279927Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:33:41.195202Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:33:52.148105Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:34:02.360682Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-31T05:34:02.750472Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:34:13.494848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:34:24.348944Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:34:35.268251Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:34:45.746982Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:34:47.702027Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T05:34:47.702065Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T05:34:47.702068Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T05:34:47.702072Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-31T05:34:47.677Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 24 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "2a8a6f65-58b8-4233-b092-c462da2bd02e" "maas.apps.be049019ee31a6cc9291.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.45:443 10.132.0.19:49534 - v1-completions-path