2026-11-03T20:33:28.518886Z info FLAG: --concurrency="0" 2026-11-03T20:33:28.519260Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T20:33:28.519308Z info FLAG: --help="false" 2026-11-03T20:33:28.519332Z info FLAG: --log_as_json="false" 2026-11-03T20:33:28.519355Z info FLAG: --log_caller="" 2026-11-03T20:33:28.519377Z info FLAG: --log_output_level="default:info" 2026-11-03T20:33:28.519399Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T20:33:28.519506Z info FLAG: --log_target="[stdout]" 2026-11-03T20:33:28.519534Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T20:33:28.519555Z info FLAG: --outlierLogPath="" 2026-11-03T20:33:28.519577Z info FLAG: --profiling="true" 2026-11-03T20:33:28.519598Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T20:33:28.519619Z info FLAG: --proxyLogLevel="warning" 2026-11-03T20:33:28.519641Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T20:33:28.519683Z info FLAG: --stsPort="0" 2026-11-03T20:33:28.519726Z info FLAG: --templateFile="" 2026-11-03T20:33:28.519754Z info FLAG: --tokenManagerPlugin="" 2026-11-03T20:33:28.519784Z info FLAG: --vklog="0" 2026-11-03T20:33:28.519810Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T20:33:28.520060Z info Proxy role ips=[10.132.1.104] type=router id=maas-default-gateway-openshift-default-855f9574cf-cmtzs.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T20:33:28.520173Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T20:33:28.522577Z info cpu limit detected as 2, setting concurrency 2026-11-03T20:33:28.522979Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T20:33:28.523056Z info JWT policy is third-party-jwt 2026-11-03T20:33:28.523084Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T20:33:28.532494Z info Starting default Istio SDS Server 2026-11-03T20:33:28.532569Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T20:33:28.532617Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T20:33:28.532831Z info Opening status port 15020 2026-11-03T20:33:28.533675Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T20:33:28.535650Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T20:33:28.536860Z info Starting proxy agent 2026-11-03T20:33:28.536945Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T20:33:28.556874Z info sds Starting SDS grpc server 2026-11-03T20:33:28.556951Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T20:33:28.789923Z info cache generated new workload certificate resourceName=default latency=249.875555ms ttl=23h59m59.210081444s 2026-11-03T20:33:28.789967Z info cache Root cert has changed, start rotating root cert 2026-11-03T20:33:28.790007Z info cache returned workload trust anchor from cache ttl=23h59m59.209993988s 2026-11-03T20:33:28.826133Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T20:33:29.294857Z info ads ADS: new connection for node:1 2026-11-03T20:33:29.295057Z info cache returned workload certificate from cache ttl=23h59m58.704945657s 2026-11-03T20:33:29.298308Z info ads ADS: new connection for node:2 2026-11-03T20:33:29.298471Z info cache returned workload trust anchor from cache ttl=23h59m58.701531076s 2026-11-03T20:33:29.625345Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:33:31.693691Z error failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused 2026-11-03T20:33:40.344788Z warn xdsproxy sendDownstreamDelta took 10.183649961s 2026-11-03T20:33:40.450246Z info Readiness succeeded in 11.94628546s 2026-11-03T20:33:40.450935Z info Envoy proxy is ready 2026-11-03T20:42:14.268347Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:42:37.576240Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:43:11.475223Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:43:15.851054Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:43:26.320598Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T20:43:30.174Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 5518 5791 89 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "31f2ff87-153e-4440-8707-208a7d07cb49" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.132.1.115:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.1.104:49342 10.132.1.104:443 10.133.1.17:42264 - v1-chat-completions-path 2026-11-03T20:43:41.881519Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T20:43:41.881549Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T20:43:41.881553Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T20:43:41.881558Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-03T20:43:41.827Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 54 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "a387455d-b1ea-4fa9-8074-c8e1c9ab70bc" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.104:443 10.133.1.17:38678 - v1-catch-all-path 2026-11-03T20:43:42.293108Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:43:43.993802Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:43:49.494154Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:44:04.723211Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:44:06.382477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:44:30.226232Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:44:41.058958Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:44:43.641107Z info ads ADS: new connection for node:3 2026-11-03T20:44:43.641525Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T20:44:43.641642Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T20:44:45.270184Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:45:25.234593Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T20:45:32.303Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 328 86 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "d8820453-8e41-4cce-adf4-a149d38ed40a" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.132.1.199:8000" outbound|54321||e2e-prio-pool-f44f3e5c-inference-pool-ip-750771d3.llm.svc.cluster.local 10.132.1.104:50156 10.132.1.104:443 10.133.1.17:57842 - v1-model-routing 2026-11-03T20:45:44.421337Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:45:44.975022Z info ads ADS: "" 3 terminated with stream closed 2026-11-03T20:45:46.299472Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:45:57.622504Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:46:08.975757Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:48:21.703253Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:48:39.057252Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:49:04.731690Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:49:16.366382Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:49:20.514436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:49:28.757802Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:49:31.517086Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:49:46.408097Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:50:30.917281Z info ads ADS: new connection for node:4 2026-11-03T20:50:30.926935Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T20:50:30.926985Z info cache read certificate from file resource=file-root:system 2026-11-03T20:50:31.464096Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:50:33.040096Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:50:36.068053Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:50:36.068172Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:50:36.068217Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:50:36.068245Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T20:50:36.219088Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T20:50:36.220852Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"288","StreamId":"12862116015220569350"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-03T20:50:35.967Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 101 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "8b755523-a6b1-4858-af5f-9d40480f9c9d" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.133.0.38:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.104:48612 10.132.1.104:443 10.133.1.17:56310 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T20:50:36.133Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 88 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "c78ce69e-a661-9fb3-b3bc-3c36782ddfd6" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.1.104:443 10.133.1.17:58076 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T20:51:21.876839Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:51:21.876871Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:51:21.876883Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:51:21.876887Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-03T20:51:21.877069Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-11-03T20:51:21.749Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1530 127 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "84c1ee42-d111-4ce2-bfad-51e52a2e303e" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.104:39686 10.132.1.104:443 10.133.1.17:34714 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T20:51:24.059995Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:51:24.060026Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:51:24.060037Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T20:51:24.060041Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:51:24.060205Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T20:51:23.936Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1536 124 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "87cc08e3-2f95-4ada-9726-3f3f626e3ddf" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.104:43058 10.132.1.104:443 10.133.1.17:34726 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T20:51:35.157964Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:51:35.157997Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:51:35.158007Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T20:51:35.158011Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:51:35.158157Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T20:51:35.061Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1529 96 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "4cb76be9-ad6b-4a80-b483-53314fec101f" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.104:43058 10.132.1.104:443 10.133.1.17:44872 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T20:52:17.884517Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:52:18.050300Z info ads ADS: "" 4 terminated 2026-11-03T20:52:18.561062Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:52:21.920608Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:52:24.027984Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:52:39.991074Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:52:51.237719Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:53:06.557336Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:53:08.354358Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:53:12.194638Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:53:19.381183Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:53:22.457329Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:53:22.457370Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:53:22.457390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:53:22.457397Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T20:53:22.386Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 77 319 71 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "d61e98ca-7958-4998-a4d2-dbbd00d8c3cd" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.133.0.38:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.104:42404 10.132.1.104:443 10.133.1.17:45090 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T20:53:24.302Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 71 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "47342e8a-8b25-4ec7-9cd1-fbf523832aa2" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.132.1.89:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.104:50158 10.132.1.104:443 10.133.1.17:45102 - v1-completions-path [2026-11-03T20:53:24.878Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 70 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "2841cde8-3b87-4829-97b3-bcec1d67b216" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.132.1.89:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.104:50162 10.132.1.104:443 10.133.1.17:45110 - v1-completions-path [2026-11-03T20:53:25.948Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 309 58 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "c7a2f35e-9db2-4d97-9608-b99913c752ef" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.132.1.89:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.104:50158 10.132.1.104:443 10.133.1.17:41038 - v1-completions-path 2026-11-03T20:53:40.880470Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:53:51.026716Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:54:02.729351Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:54:19.301418Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:54:20.970515Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:54:35.984087Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:54:38.118614Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:54:44.004585Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:54:52.418859Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:55:06.244157Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:55:36.824345Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:55:39.881341Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:56:04.763155Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:56:06.895301Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:56:07.018782Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:56:07.018817Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:56:07.018831Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:56:07.018836Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T20:56:06.927Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 89 331 91 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "f8d4970a-0330-473c-8135-b44c2848ebc3" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.133.0.38:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.104:50974 10.132.1.104:443 10.133.1.17:51902 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:56:07.543534Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:56:07.543568Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:56:07.543582Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T20:56:07.543587Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T20:56:07.482Z] "DELETE /v1/api-keys/4f51c4cc-8baa-471b-8437-f988b8befe33 HTTP/1.1" 200 - via_upstream - "-" 0 375 61 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "1f9060b7-832f-4513-a7b4-1c3db79f4051" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.133.0.38:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.104:50974 10.132.1.104:443 10.133.1.17:51906 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:56:11.168903Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:56:36.480398Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:56:56.638817Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:57:12.999978Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:57:15.639083Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:59:25.338173Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:59:35.019224Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:01:04.377972Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:01:18.566420Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:01:20.321271Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:01:25.657385Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:01:54.537086Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:01:58.542310Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T21:02:03.001Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 63 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "8e3e952b-c9fc-4368-8d3c-7baf54d0bac8" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.132.1.89:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.104:50162 10.132.1.104:443 10.133.1.17:39866 - v1-completions-path 2026-11-03T21:02:14.735878Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:02:35.005990Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:02:38.101081Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:02:54.825110Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:02:56.894141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:03:00.091162Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:03:04.233541Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 [2026-11-03T21:03:04.227Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 66 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "254bf143-9bd5-4c2e-9cdd-95aaa8511ed4" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.132.1.89:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.104:50162 10.132.1.104:443 10.133.1.17:58624 - v1-completions-path [2026-11-03T21:03:07.218Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "3cf8560f-ea61-4e74-a425-1891df74400d" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.132.1.89:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.104:50162 10.132.1.104:443 10.133.1.17:45932 - v1-completions-path 2026-11-03T21:03:21.000848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:03:31.238142Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:03:33.395565Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:03:36.639148Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:03:41.564451Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:03:58.376555Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:04:08.605266Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:04:10.257297Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:04:17.178624Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:04:20.910084Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:04:24.527357Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:04:39.327407Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:04:59.291981Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:05:09.482065Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:05:19.734913Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:05:29.884904Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:05:40.071239Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:05:50.240467Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:06:00.480438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:06:10.642912Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:06:20.851701Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:06:29.940573Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-03T21:06:31.149130Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T21:06:47.293Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 88 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "31284b8b-fbb8-4dea-8471-2a96b8c29307" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "10.134.0.36:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.104:57176 10.132.1.104:443 10.133.1.17:34262 - v1-completions-path 2026-11-03T21:06:47.714679Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-03T21:06:47.714701Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-03T21:06:47.714705Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-03T21:06:47.714708Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=26 [2026-11-03T21:06:47.689Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 25 - "44.212.242.249,10.133.1.17" "python-requests/2.34.2" "86058f47-f807-40ac-b1af-1e43850dd22f" "maas.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.104:443 10.133.1.17:34266 - v1-completions-path