2026-11-02T18:36:27.224291Z info FLAG: --concurrency="0" 2026-11-02T18:36:27.224313Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T18:36:27.224317Z info FLAG: --help="false" 2026-11-02T18:36:27.224319Z info FLAG: --log_as_json="false" 2026-11-02T18:36:27.224321Z info FLAG: --log_caller="" 2026-11-02T18:36:27.224323Z info FLAG: --log_output_level="default:info" 2026-11-02T18:36:27.224325Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T18:36:27.224329Z info FLAG: --log_target="[stdout]" 2026-11-02T18:36:27.224331Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T18:36:27.224333Z info FLAG: --outlierLogPath="" 2026-11-02T18:36:27.224335Z info FLAG: --profiling="true" 2026-11-02T18:36:27.224338Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T18:36:27.224339Z info FLAG: --proxyLogLevel="warning" 2026-11-02T18:36:27.224342Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T18:36:27.224344Z info FLAG: --stsPort="0" 2026-11-02T18:36:27.224346Z info FLAG: --templateFile="" 2026-11-02T18:36:27.224348Z info FLAG: --tokenManagerPlugin="" 2026-11-02T18:36:27.224350Z info FLAG: --vklog="0" 2026-11-02T18:36:27.224353Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T18:36:27.224520Z info Proxy role ips=[10.132.1.119] type=router id=maas-default-gateway-openshift-default-65c448b547-zws9j.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T18:36:27.224581Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T18:36:27.226165Z info cpu limit detected as 2, setting concurrency 2026-11-02T18:36:27.226496Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T18:36:27.226511Z info JWT policy is third-party-jwt 2026-11-02T18:36:27.226514Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T18:36:27.228345Z info Starting default Istio SDS Server 2026-11-02T18:36:27.228374Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T18:36:27.228400Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T18:36:27.228911Z info Opening status port 15020 2026-11-02T18:36:27.229783Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T18:36:27.230722Z info sds Starting SDS grpc server 2026-11-02T18:36:27.230742Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T18:36:27.231315Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T18:36:27.232200Z info Starting proxy agent 2026-11-02T18:36:27.232231Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T18:36:27.309100Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T18:36:27.330127Z info cache generated new workload certificate resourceName=default latency=99.231851ms ttl=23h59m59.66987813s 2026-11-02T18:36:27.330183Z info cache Root cert has changed, start rotating root cert 2026-11-02T18:36:27.330230Z info cache returned workload trust anchor from cache ttl=23h59m59.669771203s 2026-11-02T18:36:27.420134Z info ads ADS: new connection for node:1 2026-11-02T18:36:27.420211Z info cache returned workload certificate from cache ttl=23h59m59.579790797s 2026-11-02T18:36:27.420325Z info ads ADS: new connection for node:2 2026-11-02T18:36:27.420398Z info cache returned workload trust anchor from cache ttl=23h59m59.579602591s 2026-11-02T18:36:27.503655Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:36:35.623341Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:36:36.572016Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-02T18:36:36.872120Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-02T18:36:37.596335Z info Readiness succeeded in 10.374994967s 2026-11-02T18:36:37.596700Z info Envoy proxy is ready 2026-11-02T18:37:05.137546Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:20.957683Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:22.599152Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:28.028674Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:43.266050Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:44.956355Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:59.127136Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:38:09.877930Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:38:11.674352Z info ads ADS: new connection for node:3 2026-11-02T18:38:11.674571Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:38:11.674611Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:38:13.049926Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:38:53.536430Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:39:06.697582Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 [2026-11-02T18:39:11.238Z] "POST /v1/chat/completions HTTP/1.1" 403 - - "-" 131 0 51 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "a1efe75e-773e-473f-8850-a600f12f9f49" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|54321||e2e-prio-pool-3c449aaf-inference-pool-ip-02f69fdc.llm.svc.cluster.local - 10.132.1.119:443 10.132.0.15:49098 - v1-model-routing 2026-11-02T18:39:12.186335Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:39:12.620444Z info ads ADS: "" 3 terminated with stream closed 2026-11-02T18:39:14.013965Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:39:16.393763Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:39:20.723186Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:41:33.200095Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:41:49.769420Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:06.073977Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:17.771175Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:20.888194Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:29.858904Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:32.064167Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:47.052150Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:43:16.083Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 82 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "0cc4fe6c-daea-4384-8166-6984a23565cd" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.119:39550 10.132.1.119:443 10.132.0.15:38654 - v1-completions-path 2026-11-02T18:43:31.254987Z info ads ADS: new connection for node:4 2026-11-02T18:43:31.263039Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T18:43:31.264611Z info cache read certificate from file resource=file-root:system 2026-11-02T18:43:31.698604Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:43:33.255459Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:43:36.000688Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:43:36.000730Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:43:36.000746Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:43:36.000751Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:43:36.150226Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"140","StreamId":"15148372902901090886"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-02T18:43:35.885Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 115 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "c3600fc1-1bd5-4079-b8a8-4259ed458ded" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.119:47060 10.132.1.119:443 10.132.0.15:33888 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T18:43:36.077Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 73 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "a5f3e734-b99d-459b-99e6-a681e4f4ea08" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.1.119:443 10.132.0.15:33892 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-02T18:43:44.762009Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:43:44.762041Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:43:44.762051Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:43:44.762055Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:43:44.762225Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T18:43:44.633Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1717 128 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "8dcb8a05-0e24-43ba-b3f2-53eebff78886" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.119:47568 10.132.1.119:443 10.132.0.15:53600 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:44:26.690309Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:44:26.690344Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:44:26.690355Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:44:26.690359Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:44:26.690550Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T18:44:26.589Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1719 101 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "8f77e379-4328-41f2-a127-5e6ac20557ba" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.119:47568 10.132.1.119:443 10.132.0.15:47674 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:44:37.521216Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:44:37.521251Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:44:37.521261Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:44:37.521266Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:44:37.521468Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T18:44:37.420Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1719 101 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "149e65d6-398a-4c91-bd0e-3e74eef0fb85" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.119:47568 10.132.1.119:443 10.132.0.15:50466 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:45:11.737001Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-02T18:45:11.880819Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:45:11.880904Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:45:11.880928Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:45:11.880936Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T18:45:11.729Z] "DELETE /v1/api-keys/fbf4702e-5900-45b5-9f26-cbaf79bccecb HTTP/1.1" 200 - via_upstream - "-" 0 359 151 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "038ef01c-c100-4a99-aff1-faacc18fb11d" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.119:50650 10.132.1.119:443 10.132.0.15:49718 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T18:45:23.127825Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:45:23.254373Z info ads ADS: "" 4 terminated 2026-11-02T18:45:23.644189Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:45:23.644220Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:45:23.644224Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:45:23.644229Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 2026-11-02T18:45:23.749266Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:45:23.597Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 46 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "1cf7d381-ac21-43b8-8e59-12b5792be655" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.119:443 10.132.0.15:59178 - v1-completions-path 2026-11-02T18:45:26.590859Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:45:28.550834Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:45:32.650Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 56 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "a2f50b5c-32f9-42cb-aeb1-ba10d19f6924" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.119:443 10.132.0.15:59200 - v1-completions-path 2026-11-02T18:45:44.164039Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:45:45.842623Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:00.365855Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:02.084848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:04.986754Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:13.158355Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:17.459033Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-02T18:46:17.400Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 76 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "fcb60e94-ad26-473a-b648-2f56c924f245" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.119:46226 10.132.1.119:443 10.132.0.15:34976 - v1-completions-path 2026-11-02T18:46:34.405166Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:44.617708Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:46.821520Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:46:52.629Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 66 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "f2a6fc72-2651-49c2-8c95-74eb7914de1f" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.119:46226 10.132.1.119:443 10.132.0.15:38630 - v1-completions-path 2026-11-02T18:47:05.336021Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:06.957276Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:12.353573Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:23.343571Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:36.965405Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:38.512048Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:43.892939Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:02.201750Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:04.870778Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:18.872668Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:20.628477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:26.313206Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:41.332300Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:51.510414Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:49:06.889240Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:49:09.215535Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:51:22.236826Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:51:24.629247Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:52:27.290334Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:52:27.290361Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:52:27.290364Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:52:27.290368Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T18:52:27.242Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 403 - - "-" 73 113 48 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "b7770dfe-c191-490c-ba6e-6ceef947d699" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.119:443 10.132.0.15:34550 - v1-completions-path 2026-11-02T18:52:49.019157Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:52:49.516Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 429 - - "-" 73 18 90 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "39de7aae-ff98-40c5-810e-304d44ad2b46" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.119:443 10.132.0.15:52786 - v1-completions-path 2026-11-02T18:53:04.030401Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:05.573472Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:11.413800Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:30.194368Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:34.004834Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:49.215908Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:59.629342Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:02.687747Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:19.537609Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:21.775718Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:25.161645Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:29.676194Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:54:29.676232Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:54:29.676248Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T18:54:29.676253Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T18:54:29.604Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 66 308 71 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "17117bef-64ab-48df-8c07-c27cfe44e6bd" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.1.153:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.119:51700 10.132.1.119:443 10.132.0.15:54438 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T18:54:33.912Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 64 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "16ae31d4-049f-4599-b4fd-ea05d6365e3f" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.119:46226 10.132.1.119:443 10.132.0.15:49304 - v1-completions-path 2026-11-02T18:54:47.109487Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:57.287077Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:59.290087Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:02.524913Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:07.822756Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:23.988093Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:34.164493Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:35.695007Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:42.999093Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:45.893300Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:49.167927Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:05.131247Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:25.006360Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:35.266508Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:45.533497Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:55.744413Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:05.876791Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:16.164148Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:26.299353Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:36.593708Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:46.781740Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:57.023170Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:58:06.779942Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:58:06.779966Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:58:06.779970Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:58:06.779974Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T18:58:06.745Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 34 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "b7445e0a-a28c-462b-8afd-dd48a4c3ac30" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.119:443 10.132.0.15:49062 - v1-completions-path