2026-11-03T01:25:26.303076Z info FLAG: --concurrency="0" 2026-11-03T01:25:26.303095Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T01:25:26.303098Z info FLAG: --help="false" 2026-11-03T01:25:26.303101Z info FLAG: --log_as_json="false" 2026-11-03T01:25:26.303103Z info FLAG: --log_caller="" 2026-11-03T01:25:26.303105Z info FLAG: --log_output_level="default:info" 2026-11-03T01:25:26.303107Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T01:25:26.303111Z info FLAG: --log_target="[stdout]" 2026-11-03T01:25:26.303113Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T01:25:26.303115Z info FLAG: --outlierLogPath="" 2026-11-03T01:25:26.303117Z info FLAG: --profiling="true" 2026-11-03T01:25:26.303119Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T01:25:26.303121Z info FLAG: --proxyLogLevel="warning" 2026-11-03T01:25:26.303122Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T01:25:26.303124Z info FLAG: --stsPort="0" 2026-11-03T01:25:26.303126Z info FLAG: --templateFile="" 2026-11-03T01:25:26.303128Z info FLAG: --tokenManagerPlugin="" 2026-11-03T01:25:26.303130Z info FLAG: --vklog="0" 2026-11-03T01:25:26.303133Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T01:25:26.303275Z info Proxy role ips=[10.134.0.149] type=router id=maas-default-gateway-openshift-default-7bdc57c4cf-bd9fs.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T01:25:26.303319Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T01:25:26.304407Z info cpu limit detected as 2, setting concurrency 2026-11-03T01:25:26.304641Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T01:25:26.304651Z info JWT policy is third-party-jwt 2026-11-03T01:25:26.304654Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T01:25:26.306987Z info Starting default Istio SDS Server 2026-11-03T01:25:26.307012Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T01:25:26.307032Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T01:25:26.307178Z info Opening status port 15020 2026-11-03T01:25:26.307815Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T01:25:26.308032Z info sds Starting SDS grpc server 2026-11-03T01:25:26.308049Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T01:25:26.309045Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T01:25:26.310180Z info Starting proxy agent 2026-11-03T01:25:26.310216Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T01:25:26.371692Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T01:25:26.487358Z info cache generated new workload certificate resourceName=default latency=179.228626ms ttl=23h59m59.512645905s 2026-11-03T01:25:26.487396Z info cache Root cert has changed, start rotating root cert 2026-11-03T01:25:26.487426Z info cache returned workload trust anchor from cache ttl=23h59m59.512574242s 2026-11-03T01:25:26.517279Z info ads ADS: new connection for node:1 2026-11-03T01:25:26.517346Z info cache returned workload certificate from cache ttl=23h59m59.482656735s 2026-11-03T01:25:26.517798Z info ads ADS: new connection for node:2 2026-11-03T01:25:26.517896Z info cache returned workload trust anchor from cache ttl=23h59m59.482105177s 2026-11-03T01:25:26.601041Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:25:34.306652Z info Readiness succeeded in 8.005680126s 2026-11-03T01:25:34.307517Z info Envoy proxy is ready 2026-11-03T01:25:36.327129Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:25:37.046346Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:25:37.046395Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:25:37.046417Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:25:37.046422Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T01:25:36.882Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 317 164 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "56566da6-deec-4206-b800-087c5229f3ce" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.149:60194 10.134.0.149:443 10.132.0.14:56032 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T01:25:39.624485Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T01:25:39.545Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 79 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "1231340e-55db-44ad-b869-a46bc706f46b" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:47742 10.134.0.149:443 10.132.0.14:56042 - v1-catch-all-path 2026-11-03T01:25:50.139469Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:26:00.342923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:26:17.034018Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:26:28.724509Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:26:32.520709Z info ads ADS: new connection for node:3 2026-11-03T01:26:32.520909Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T01:26:32.520990Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T01:26:35.598261Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:27:14.688236Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:27:20.691Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 349 87 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "ca349824-7475-4c52-b9c6-398dfe0eeaf3" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.132.1.113:8000" outbound|54321||e2e-prio-pool-a2af2b5f-inference-pool-ip-665c27da.llm.svc.cluster.local 10.134.0.149:43252 10.134.0.149:443 10.132.0.14:57454 - v1-model-routing [2026-11-03T01:27:20.981Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 345 54 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "0b0e0830-0392-4f81-b832-3e73a6526a5d" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.132.1.113:8000" outbound|54321||e2e-prio-pool-a2af2b5f-inference-pool-ip-665c27da.llm.svc.cluster.local 10.134.0.149:43252 10.134.0.149:443 10.132.0.14:57456 - v1-model-routing 2026-11-03T01:27:38.730214Z info ads ADS: "" 3 terminated with stream closed 2026-11-03T01:27:38.787348Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:27:45.218578Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:27:58.341716Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:30:13.720585Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:30:13.720621Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:30:13.720633Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:30:13.720638Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T01:30:13.651Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 68 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "9f0c4510-aa98-43ae-916c-bc2cd2eb2a04" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.149:38088 10.134.0.149:443 10.132.0.14:41236 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T01:30:32.262492Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:31:00.268128Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:31:26.158291Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:31:38.208355Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:31:44.596062Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:31:51.895761Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:32:03.096962Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:32:09.279197Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:32:27.236403Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:33:13.010Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 87 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "d92845e4-a077-49a1-848f-3c511e130d24" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:39872 10.134.0.149:443 10.132.0.14:53750 - v1-completions-path 2026-11-03T01:33:14.793467Z info ads ADS: new connection for node:4 2026-11-03T01:33:14.798154Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T01:33:14.798198Z info cache read certificate from file resource=file-root:system 2026-11-03T01:33:18.694606Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:33:19.922764Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:33:29.905878Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T01:33:29.906408Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"157","StreamId":"17700130333097849041"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-03T01:33:29.846Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 59 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "63bfea47-17c7-4e90-8eba-ec182ab83c65" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.134.0.149:443 10.132.0.14:46590 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T01:33:50.878269Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:33:50.878302Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:33:50.878314Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:33:50.878318Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T01:33:50.878467Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T01:33:50.751Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1714 126 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "c327dcc3-8010-4e66-9a2f-3f135daeaf3b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.149:39442 10.134.0.149:443 10.132.0.14:42772 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T01:35:08.758509Z info ads ADS: "" 4 terminated with stream closed 2026-11-03T01:35:08.804872Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:35:11.470555Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:35:22.089041Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:35:24.971Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 77 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "d0cc552e-68cf-46b3-aa54-364c3ed1ed25" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:52934 10.134.0.149:443 10.132.0.14:35278 - v1-completions-path 2026-11-03T01:35:40.442678Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:35:45.097966Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:35:48.412Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 61 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "c182ded0-5c58-4066-a2cf-27f2c7c6bdfb" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:52934 10.134.0.149:443 10.132.0.14:48434 - v1-completions-path [2026-11-03T01:35:49.477Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 299 62 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "e4ba1a58-d1d6-49bd-8b5f-36aa3f844572" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:44564 10.134.0.149:443 10.132.0.14:48444 - v1-completions-path [2026-11-03T01:35:50.014Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 289 66 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "db11b7cc-fe6f-4ea6-ba1c-e8cc84314610" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:52934 10.134.0.149:443 10.132.0.14:48448 - v1-completions-path 2026-11-03T01:35:50.871754Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-03T01:35:50.554Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 56 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "8938fe37-0529-4d1b-a4d0-0c76ae35b455" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:52934 10.134.0.149:443 10.132.0.14:48472 - v1-completions-path [2026-11-03T01:35:50.812Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 62 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "02231a5a-0114-49ca-b004-6cb3897eef29" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:44564 10.134.0.149:443 10.132.0.14:48484 - v1-completions-path 2026-11-03T01:36:05.171346Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:36:21.978690Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:36:28.275Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 63 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "40d7c0d5-d48c-4740-a451-5c7da4e60342" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:44564 10.134.0.149:443 10.132.0.14:36094 - v1-completions-path 2026-11-03T01:36:49.815493Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:37:00.386300Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:37:06.813279Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:37:18.138Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 58 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "0c0d9833-5206-426a-81ff-424f4ee5db2b" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:52934 10.134.0.149:443 10.132.0.14:51820 - v1-completions-path [2026-11-03T01:37:19.710Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 61 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "d3ccf2af-e265-4b07-901a-b3c85bc1e8a6" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:44564 10.134.0.149:443 10.132.0.14:51822 - v1-completions-path [2026-11-03T01:37:19.974Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 55 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "eeefe1ed-4c2f-4413-a535-88f864b43a25" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:44564 10.134.0.149:443 10.132.0.14:51838 - v1-completions-path 2026-11-03T01:37:35.203685Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:37:39.814423Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:37:53.235073Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:38:06.246902Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:38:14.103061Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:38:24.241098Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:38:34.762901Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:38:52.294692Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:38:55.186Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 66 18 58 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "a513d084-65c0-4bb0-a367-190c10c93bc1" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.149:443 10.132.0.14:60942 - v1-completions-path 2026-11-03T01:38:58.253236Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:39:16.380896Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:39:21.904238Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:39:23.448Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 33 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "828e24c1-b11e-4db1-baf5-e55f75c32465" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.149:443 10.132.0.14:55514 - v1-completions-path 2026-11-03T01:39:24.112801Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:39:24.112871Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:39:24.112893Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T01:39:24.112900Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T01:39:24.000Z] "DELETE /v1/api-keys/31658a78-3d4d-4110-8519-e6bb7961857a HTTP/1.1" 200 - via_upstream - "-" 0 375 112 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "21bda3d2-a974-4ed6-a4c4-d784612f38a0" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.149:34012 10.134.0.149:443 10.132.0.14:55526 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T01:39:33.173590Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:39:56.548214Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:40:07.245948Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:40:25.966254Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:40:33.680951Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:42:59.790461Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:44:29.071374Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:44:31.442Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 429 - - "-" 73 18 107 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "968b4058-0bbf-4de8-9f49-514835b8e98b" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.149:443 10.132.0.14:49116 - v1-completions-path 2026-11-03T01:44:48.367551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:44:53.148404Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:44:56.274747Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T01:44:56.274775Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T01:44:56.274778Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T01:44:56.274782Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-03T01:44:56.240Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 34 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "ba21fc55-81ba-412a-b8c8-68eeb2b25be0" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.149:443 10.132.0.14:36560 - v1-completions-path 2026-11-03T01:45:06.518962Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:45:22.653001Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:45:27.479145Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:45:32.548Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 63 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "b5d4dfaf-3ff9-4f74-aa34-2c5118aef272" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.149:443 10.132.0.14:45350 - v1-completions-path 2026-11-03T01:45:47.159351Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:45:57.984101Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:46:04.285792Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:46:18.506Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 70 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "6983ae38-f9cc-4a8d-8cb9-ef7a834ceffd" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:44564 10.134.0.149:443 10.132.0.14:59286 - v1-completions-path 2026-11-03T01:46:35.245532Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:46:39.696017Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:46:47.332588Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:46:56.351Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 66 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "07b76277-e30b-4a4c-b407-fc6bdf5b12cc" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:52934 10.134.0.149:443 10.132.0.14:42648 - v1-completions-path [2026-11-03T01:47:04.013Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 58 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "34f8e674-497f-4a9d-b787-41c18986adcb" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.149:443 10.132.0.14:42664 - v1-completions-path 2026-11-03T01:47:18.409848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:47:28.979886Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:47:35.596780Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:47:41.466431Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:47:53.615572Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T01:48:00.436Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 68 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "53582f88-9bfe-4700-be49-4595169f7ec5" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:44564 10.134.0.149:443 10.132.0.14:41804 - v1-completions-path [2026-11-03T01:48:02.289Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "068b560e-9dc9-4347-833a-a1fcf9627bbe" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.149:44564 10.134.0.149:443 10.132.0.14:41812 - v1-completions-path 2026-11-03T01:48:17.073177Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:48:27.732689Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:48:32.492960Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:48:50.905069Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:48:56.900096Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:49:20.796575Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:49:42.994133Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:50:02.835473Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:50:15.624021Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:50:35.653721Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:50:46.468504Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:50:57.184459Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:51:08.030701Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T01:51:18.620762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9