2026-10-31T02:02:44.775654Z info FLAG: --concurrency="0" 2026-10-31T02:02:44.775672Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-31T02:02:44.775676Z info FLAG: --help="false" 2026-10-31T02:02:44.775678Z info FLAG: --log_as_json="false" 2026-10-31T02:02:44.775680Z info FLAG: --log_caller="" 2026-10-31T02:02:44.775682Z info FLAG: --log_output_level="default:info" 2026-10-31T02:02:44.775684Z info FLAG: --log_stacktrace_level="default:none" 2026-10-31T02:02:44.775689Z info FLAG: --log_target="[stdout]" 2026-10-31T02:02:44.775691Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-31T02:02:44.775693Z info FLAG: --outlierLogPath="" 2026-10-31T02:02:44.775695Z info FLAG: --profiling="true" 2026-10-31T02:02:44.775697Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-31T02:02:44.775699Z info FLAG: --proxyLogLevel="warning" 2026-10-31T02:02:44.775701Z info FLAG: --serviceCluster="istio-proxy" 2026-10-31T02:02:44.775703Z info FLAG: --stsPort="0" 2026-10-31T02:02:44.775705Z info FLAG: --templateFile="" 2026-10-31T02:02:44.775706Z info FLAG: --tokenManagerPlugin="" 2026-10-31T02:02:44.775709Z info FLAG: --vklog="0" 2026-10-31T02:02:44.775711Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-31T02:02:44.775842Z info Proxy role ips=[10.133.0.239] type=router id=maas-default-gateway-openshift-default-848f6bf8bc-fr8bl.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-31T02:02:44.775881Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-31T02:02:44.776871Z info cpu limit detected as 2, setting concurrency 2026-10-31T02:02:44.777132Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-31T02:02:44.777144Z info JWT policy is third-party-jwt 2026-10-31T02:02:44.777147Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-31T02:02:44.779077Z info Starting default Istio SDS Server 2026-10-31T02:02:44.779101Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-31T02:02:44.779125Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-31T02:02:44.779490Z info Opening status port 15020 2026-10-31T02:02:44.780268Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-31T02:02:44.781135Z info sds Starting SDS grpc server 2026-10-31T02:02:44.781251Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-31T02:02:44.781687Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-31T02:02:44.782641Z info Starting proxy agent 2026-10-31T02:02:44.782672Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-31T02:02:44.843671Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-31T02:02:44.929633Z info cache generated new workload certificate resourceName=default latency=148.300618ms ttl=23h59m59.070370602s 2026-10-31T02:02:44.929664Z info cache Root cert has changed, start rotating root cert 2026-10-31T02:02:44.929694Z info cache returned workload trust anchor from cache ttl=23h59m59.070306975s 2026-10-31T02:02:44.989671Z info ads ADS: new connection for node:1 2026-10-31T02:02:44.989756Z info ads ADS: new connection for node:2 2026-10-31T02:02:44.989766Z info cache returned workload trust anchor from cache ttl=23h59m59.010235341s 2026-10-31T02:02:44.989829Z info cache returned workload certificate from cache ttl=23h59m59.010174986s 2026-10-31T02:02:45.079664Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:02:53.181363Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:02:54.273672Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-10-31T02:02:55.103363Z info Readiness succeeded in 10.329947402s 2026-10-31T02:02:55.103784Z info Envoy proxy is ready 2026-10-31T02:03:00.410467Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:03:16.742957Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:03:28.334684Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:03:31.009742Z info ads ADS: new connection for node:3 2026-10-31T02:03:31.009943Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-31T02:03:31.009976Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-31T02:03:33.337970Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:04:19.562848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:04:22.561615Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:04:22.561654Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:04:22.561669Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T02:04:22.561674Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T02:04:22.421Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 140 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "15f6dcb4-2270-4bd8-8901-0c4fd464282b" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.133.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.239:55890 10.133.0.239:443 10.132.0.9:52908 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T02:04:24.557Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 336 87 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "f89ffc22-ae4a-41a4-91eb-cbdddba93b48" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.132.0.213:8000" outbound|54321||e2e-prio-pool-91b83e68-inference-pool-ip-9ebc5b1f.llm.svc.cluster.local 10.133.0.239:54268 10.133.0.239:443 10.132.0.9:52934 - v1-model-routing [2026-10-31T02:04:26.032Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 345 59 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "acb627bd-2569-4cf7-a842-2dba86271acd" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.132.0.213:8000" outbound|54321||e2e-prio-pool-91b83e68-inference-pool-ip-9ebc5b1f.llm.svc.cluster.local 10.133.0.239:54282 10.133.0.239:443 10.132.0.9:52948 - v1-model-routing 2026-10-31T02:04:43.510112Z info ads ADS: "" 3 terminated with stream closed 2026-10-31T02:04:43.783345Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:04:48.616386Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:05:05.544622Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:07:41.358270Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:08:11.371881Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:08:41.314818Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:08:53.251835Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:08:59.415954Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:09:15.380688Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:09:26.556362Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:09:32.672104Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:09:52.046283Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:10:39.916307Z info ads ADS: new connection for node:4 2026-10-31T02:10:39.921872Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-31T02:10:39.921925Z info cache read certificate from file resource=file-root:system 2026-10-31T02:10:43.846086Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:10:44.906897Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:10:47.731151Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:10:47.731193Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:10:47.731214Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T02:10:47.731220Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T02:10:47.614Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 116 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "abb9529b-0d4c-471d-bfd9-3d97ead6d3bd" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.133.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.239:59382 10.133.0.239:443 10.132.0.9:54928 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T02:10:50.253902Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"158","StreamId":"2228486159568835521"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=26 [2026-10-31T02:10:50.193Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 60 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "516834f5-a290-4a67-aceb-1796663c437a" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.239:443 10.132.0.9:54936 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-31T02:12:04.030917Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T02:12:04.030959Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T02:12:04.030980Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-31T02:12:04.030987Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-10-31T02:12:03.914Z] "DELETE /v1/api-keys/3d9b323e-1028-4f2f-b19f-dae5c1de8813 HTTP/1.1" 200 - via_upstream - "-" 0 359 116 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "86948d64-cbd5-4da0-bb8b-f4dd603ee340" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.133.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.239:58696 10.133.0.239:443 10.132.0.9:45398 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T02:12:17.327061Z info ads ADS: "" 4 terminated with stream closed 2026-10-31T02:12:17.398003Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:12:19.928652Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:12:30.513540Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:12:48.823344Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:12:53.591988Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:13:13.793099Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:13:30.758303Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:13:40.817403Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 [2026-10-31T02:13:40.746Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 72 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "b8390271-ae0e-4fd8-9369-15a670463e77" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58858 10.133.0.239:443 10.132.0.9:57954 - v1-completions-path 2026-10-31T02:13:41.572927Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-31T02:13:41.514Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 59 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "3fda9ed5-034e-4e0b-8363-aeb6ba48e160" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58874 10.133.0.239:443 10.132.0.9:57970 - v1-completions-path [2026-10-31T02:13:43.345Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 60 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "c57cea70-3c87-4921-91df-2082f43a5ef9" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58858 10.133.0.239:443 10.132.0.9:57982 - v1-completions-path 2026-10-31T02:14:01.650319Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:14:12.486112Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:14:18.812771Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T02:14:30.158Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 59 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "ae8f36a9-d4b2-400b-ab7b-eac06e2c535c" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58858 10.133.0.239:443 10.132.0.9:40428 - v1-completions-path [2026-10-31T02:14:30.933Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 93 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "c94e57b0-6eb0-4ff5-bf91-adbabd583db9" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58874 10.133.0.239:443 10.132.0.9:40430 - v1-completions-path [2026-10-31T02:14:32.512Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 56 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "b67625cc-a997-48e6-a434-4a47b3a85741" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58858 10.133.0.239:443 10.132.0.9:40446 - v1-completions-path 2026-10-31T02:14:47.009196Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:14:51.651401Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:15:05.138619Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:15:15.231643Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:15:25.867705Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:15:32.363038Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:15:47.119248Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:16:00.417583Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:16:06.404581Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:16:24.464796Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:16:29.921800Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:16:41.272737Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:16:53.522261Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:16:53.522304Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:16:53.522321Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T02:16:53.522326Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T02:16:53.407Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 309 114 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "73b33917-a024-4d83-a28d-d70ceeb65566" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.133.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.239:41602 10.133.0.239:443 10.132.0.9:38332 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T02:16:54.046Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 59 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "d32b33b5-2280-4bfd-a5d0-ee38b50db292" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58858 10.133.0.239:443 10.132.0.9:38346 - v1-completions-path 2026-10-31T02:17:08.083473Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:17:18.800584Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:17:37.370537Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:17:45.116826Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:20:46.732343Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:22:15.136644Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T02:22:19.893Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 73 292 79 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "80b728bc-e1de-4a32-9304-c57831c266e9" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.132.0.56:8000" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:46140 10.133.0.239:443 10.132.0.9:49534 - v1-completions-path 2026-10-31T02:22:20.968600Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:22:20.968635Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:22:20.968648Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T02:22:20.968652Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T02:22:20.859Z] "DELETE /v1/api-keys/e2e-trlp-test-key HTTP/1.1" 404 - via_upstream - "-" 0 29 109 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "4ac1e303-cf0f-4193-bb8c-5d779f74c0b5" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.133.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.239:45770 10.133.0.239:443 10.132.0.9:49544 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T02:22:34.501241Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:22:39.467218Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:22:52.460210Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:23:09.070438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:23:14.043762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:23:33.808964Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:23:44.573854Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:23:50.862625Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T02:24:04.513Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 62 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "a9851844-2dec-4581-8266-69676ae9732d" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58858 10.133.0.239:443 10.132.0.9:47164 - v1-completions-path [2026-10-31T02:24:05.457Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 58 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "68464ce5-273a-4147-9d59-864c86beac89" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58874 10.133.0.239:443 10.132.0.9:47174 - v1-completions-path 2026-10-31T02:24:07.146328Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T02:24:07.146367Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T02:24:07.146392Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T02:24:07.146398Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-10-31T02:24:06.781Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 59 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "c2f81e55-2adc-4b90-a543-5d26a08405d5" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58874 10.133.0.239:443 10.132.0.9:55346 - v1-completions-path [2026-10-31T02:24:07.033Z] "DELETE /v1/api-keys/b87142f5-46a7-4470-96af-6f1649379b01 HTTP/1.1" 200 - via_upstream - "-" 0 365 113 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "c88e1f62-3766-4f76-bffd-6e11ad9cf705" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.1.70:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.239:33748 10.133.0.239:443 10.132.0.9:55360 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T02:24:21.011984Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:24:27.293782Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:24:33.432217Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:24:38.641258Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:24:38.641295Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:24:38.641310Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T02:24:38.641315Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T02:24:38.520Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 66 308 120 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "8d4ba611-5330-408c-b4be-d7cc92fbac01" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.132.0.250:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.239:42860 10.133.0.239:443 10.132.0.9:38678 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T02:24:43.624Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "3265dc1f-3f16-4145-a84e-38eed6dadec0" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58858 10.133.0.239:443 10.132.0.9:38700 - v1-completions-path [2026-10-31T02:24:43.984Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 55 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "e2175205-2877-4cd6-a2f8-102c8688f751" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58874 10.133.0.239:443 10.132.0.9:38710 - v1-completions-path 2026-10-31T02:25:02.657537Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:25:13.387829Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:25:19.933668Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:25:26.047727Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T02:25:31.355Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 65 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "aaf2ef6e-67b4-4409-9b52-533ae9c65e5f" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:58874 10.133.0.239:443 10.132.0.9:47208 - v1-completions-path 2026-10-31T02:25:35.273225Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:25:58.211841Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:26:08.861380Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:26:13.810532Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:26:31.631178Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:26:39.552687Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:27:01.843167Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:27:23.814157Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:27:43.973987Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:27:54.809082Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:28:05.835816Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:28:16.648152Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:28:27.493859Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:28:38.353466Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:28:49.073706Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T02:28:59.178633Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T02:28:59.178670Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T02:28:59.178686Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T02:28:59.178692Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-31T02:28:59.678266Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T02:28:59.059Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 330 119 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "8bdd93b6-9d2e-4dc2-a9c6-646b6264154d" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.1.70:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.239:35236 10.133.0.239:443 10.132.0.9:52404 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T02:29:10.504Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 79 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "07f14885-af9f-4890-9a86-7cc84cf2ff74" "maas.apps.5ce226450bd4aaa499fe.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.239:53236 10.133.0.239:443 10.132.0.9:35842 - v1-completions-path