2026-11-01T00:28:56.969269Z info FLAG: --concurrency="0" 2026-11-01T00:28:56.969291Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-01T00:28:56.969297Z info FLAG: --help="false" 2026-11-01T00:28:56.969300Z info FLAG: --log_as_json="false" 2026-11-01T00:28:56.969303Z info FLAG: --log_caller="" 2026-11-01T00:28:56.969306Z info FLAG: --log_output_level="default:info" 2026-11-01T00:28:56.969308Z info FLAG: --log_stacktrace_level="default:none" 2026-11-01T00:28:56.969314Z info FLAG: --log_target="[stdout]" 2026-11-01T00:28:56.969318Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-01T00:28:56.969321Z info FLAG: --outlierLogPath="" 2026-11-01T00:28:56.969324Z info FLAG: --profiling="true" 2026-11-01T00:28:56.969327Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-01T00:28:56.969331Z info FLAG: --proxyLogLevel="warning" 2026-11-01T00:28:56.969333Z info FLAG: --serviceCluster="istio-proxy" 2026-11-01T00:28:56.969337Z info FLAG: --stsPort="0" 2026-11-01T00:28:56.969340Z info FLAG: --templateFile="" 2026-11-01T00:28:56.969342Z info FLAG: --tokenManagerPlugin="" 2026-11-01T00:28:56.969346Z info FLAG: --vklog="0" 2026-11-01T00:28:56.969350Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-01T00:28:56.969531Z info Proxy role ips=[10.134.1.160] type=router id=maas-default-gateway-openshift-default-6cc458d887-w2bvl.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-01T00:28:56.969589Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-01T00:28:56.970725Z info cpu limit detected as 2, setting concurrency 2026-11-01T00:28:56.970969Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-01T00:28:56.970984Z info JWT policy is third-party-jwt 2026-11-01T00:28:56.970988Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-01T00:28:56.972667Z info Starting default Istio SDS Server 2026-11-01T00:28:56.972694Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-01T00:28:56.972712Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-01T00:28:56.973108Z info Opening status port 15020 2026-11-01T00:28:56.973977Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-01T00:28:56.975348Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-01T00:28:56.976280Z info Starting proxy agent 2026-11-01T00:28:56.976321Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-01T00:28:56.984082Z info sds Starting SDS grpc server 2026-11-01T00:28:56.984106Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-01T00:28:57.032012Z info cache generated new workload certificate resourceName=default latency=58.161881ms ttl=23h59m59.967992758s 2026-11-01T00:28:57.032056Z info cache Root cert has changed, start rotating root cert 2026-11-01T00:28:57.032102Z info cache returned workload trust anchor from cache ttl=23h59m59.967898797s 2026-11-01T00:28:57.043539Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-01T00:28:57.168277Z info ads ADS: new connection for node:1 2026-11-01T00:28:57.168355Z info cache returned workload certificate from cache ttl=23h59m59.831648122s 2026-11-01T00:28:57.168596Z info ads ADS: new connection for node:2 2026-11-01T00:28:57.168670Z info cache returned workload trust anchor from cache ttl=23h59m59.831330823s 2026-11-01T00:28:57.254014Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:29:05.341446Z info Readiness succeeded in 8.375960214s 2026-11-01T00:29:05.341874Z info Envoy proxy is ready 2026-11-01T00:29:23.998263Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:29:26.660535Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:29:26.660682Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:29:26.660776Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:29:26.660820Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T00:29:26.515Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 61 304 145 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "4602f922-a94c-4983-bcc6-3c59e989c06d" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.160:47870 10.134.1.160:443 10.132.0.9:38684 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-01T00:29:27.713Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 140 405 81 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "204e1c82-dddd-98b0-be79-a3d78a44f0e3" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.0.44:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:48990 10.134.1.160:443 10.132.0.9:38694 - v1-chat-completions-path 2026-11-01T00:29:39.779930Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:29:42.210830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:29:48.596880Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:29:52.120Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 429 - - "-" 61 18 60 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "db706fe0-0fc5-40bc-bb5e-e4fd7633af86" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.160:443 10.132.0.9:39874 - v1-catch-all-path 2026-11-01T00:29:55.698780Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:30:05.592972Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:30:20.398720Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:30:31.411912Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:30:33.410704Z info ads ADS: new connection for node:3 2026-11-01T00:30:33.411031Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T00:30:33.411119Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T00:30:34.488188Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:31:15.902056Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:31:19.036192Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:31:19.036232Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:31:19.036247Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:31:19.036253Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T00:31:18.958Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 77 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "a99a9ac3-656e-4e5f-8cdd-cd38c28184cd" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.160:34416 10.134.1.160:443 10.132.0.9:58414 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-01T00:31:22.600Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 346 85 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "13756e04-da93-4d04-b85c-7e3ccf605927" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.1.164:8000" outbound|54321||e2e-prio-pool-ca03fe61-inference-pool-ip-13923081.llm.svc.cluster.local 10.134.1.160:54526 10.134.1.160:443 10.132.0.9:49630 - v1-model-routing 2026-11-01T00:31:35.320410Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:31:35.349245Z info ads ADS: "" 3 terminated with stream closed 2026-11-01T00:31:37.726304Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:31:40.558152Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:31:45.617653Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:34:05.265390Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:34:21.838351Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:34:38.342775Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:34:50.313626Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:34:54.127072Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:35:05.933937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:35:09.456326Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:35:12.178833Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:35:12.178886Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:35:12.178912Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:35:12.178921Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T00:35:12.089Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 93 336 89 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "4befc810-1238-40c0-a178-97c5c957d830" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.160:34354 10.134.1.160:443 10.132.0.9:58540 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T00:35:25.145263Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:36:12.258004Z info ads ADS: new connection for node:4 2026-11-01T00:36:12.263048Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-01T00:36:12.263104Z info cache read certificate from file resource=file-root:system 2026-11-01T00:36:13.430239Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:36:14.983558Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:36:17.546079Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:36:17.546133Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:36:17.546157Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:36:17.546165Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T00:36:15.731Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 1814 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "722385c1-a8be-4cf7-a8a7-2c38781440ca" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.133.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.160:54504 10.134.1.160:443 10.132.0.9:56488 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T00:36:21.953149Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"146","StreamId":"16188926200847887910"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=26 [2026-11-01T00:36:21.890Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 62 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "a38f1c64-fc77-4adf-a3a7-88989ced65d0" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.134.1.160:443 10.132.0.9:57726 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-01T00:36:32.605200Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-01T00:36:32.605566Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:36:32.605591Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:36:32.605604Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T00:36:32.605611Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:36:32.605821Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-01T00:36:32.489Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1719 116 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "8e356ea8-3d0f-422a-b45d-537a793a7b9b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.1.160:59972 10.134.1.160:443 10.132.0.9:41428 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-01T00:37:19.156779Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:37:19.156814Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:37:19.156826Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:37:19.156831Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-01T00:37:19.157032Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-11-01T00:37:19.050Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1716 106 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "e5b46aea-0d55-4f8d-aa64-5ab45e1cafab" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.1.160:36726 10.134.1.160:443 10.132.0.9:41478 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-01T00:37:56.816251Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:37:56.936037Z info ads ADS: "" 4 terminated with stream closed 2026-11-01T00:37:58.317951Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:38:05.115502Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:38:20.732787Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:38:25.260148Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:38:28.329587Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 [2026-11-01T00:38:28.275Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 74 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "0fecc62a-8b0d-456a-b6b7-57de571275bc" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:55018 10.134.1.160:443 10.132.0.9:56218 - v1-completions-path 2026-11-01T00:38:40.540389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:38:56.601421Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:39:02.337Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 62 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "75ec9e8f-89b5-4b97-bf42-a44c72666086" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:55018 10.134.1.160:443 10.132.0.9:55816 - v1-completions-path 2026-11-01T00:39:19.998362Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:39:30.314396Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:39:34.888528Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:39:38.448Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 64 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "6737ee2c-6ab5-48a0-a63a-d0839de59983" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:55018 10.134.1.160:443 10.132.0.9:44802 - v1-completions-path [2026-11-01T00:39:38.723Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 62 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "7d1c3bc0-41fc-4fb5-8794-a5d03166f8c2" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:37708 10.134.1.160:443 10.132.0.9:44806 - v1-completions-path [2026-11-01T00:39:39.107Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 57 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "7f373776-99d3-4f96-b8f0-fee96e1eba10" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:55018 10.134.1.160:443 10.132.0.9:44808 - v1-completions-path [2026-11-01T00:39:39.268Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 55 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "d9ce9113-ea9c-4c11-9da2-ab4b07c38071" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:37708 10.134.1.160:443 10.132.0.9:44812 - v1-completions-path 2026-11-01T00:39:51.853678Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:39:54.176412Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:01.302393Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:13.212717Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:18.408622Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:40:24.260Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 78 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "8a1bcb3a-a6bb-4756-b4e5-31b335ceb956" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:45290 10.134.1.160:443 10.132.0.9:52624 - v1-completions-path 2026-11-01T00:40:26.746734Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:31.971200Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:50.288933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:40:53.245638Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:40:56.450Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 281 69 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "a890efab-dfa8-4fc7-b779-ed256bcde251" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:45290 10.134.1.160:443 10.132.0.9:38150 - v1-completions-path 2026-11-01T00:41:17.121840Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:41:35.076463Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:41:45.356206Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:45:26.204631Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:45:41.559316Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:45:43.857174Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:45:46.694452Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-01T00:45:46.694480Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-01T00:45:46.694483Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-01T00:45:46.694488Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=26 [2026-11-01T00:45:44.680Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 2014 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "f97af07a-88a5-4875-acc5-9044249a4e48" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.160:443 10.132.0.9:36390 - v1-completions-path 2026-11-01T00:45:51.010006Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:46:10.768075Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:46:14.384504Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:46:17.284519Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:46:17.284563Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T00:46:17.284588Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T00:46:17.284596Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-01T00:46:17.193Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 315 91 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "b0a74893-9aae-45ce-aa9a-fabeaecc3067" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.1.112:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.160:48400 10.134.1.160:443 10.132.0.9:42510 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T00:46:29.963274Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:46:40.371113Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:46:45.474776Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:46:49.467Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "d0d864bc-f784-4fd6-9979-3312870205ed" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:55018 10.134.1.160:443 10.132.0.9:55020 - v1-completions-path 2026-11-01T00:47:02.453194Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:06.895774Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:10.428361Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T00:47:16.591Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 65 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "94e3abce-cb36-4590-8d7d-fc6c46332672" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:37708 10.134.1.160:443 10.132.0.9:56990 - v1-completions-path [2026-11-01T00:47:18.397Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 57 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "51139f79-48e4-4723-be55-279ba1a6cb43" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.134.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.160:55018 10.134.1.160:443 10.132.0.9:56992 - v1-completions-path 2026-11-01T00:47:33.135935Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:43.497029Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:47.921437Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:51.362300Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:47:56.308771Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:14.794417Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:25.101538Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:27.322317Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:36.917657Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:40.528330Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:48:58.359112Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:49:00.587912Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:49:19.034380Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:49:29.522049Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:49:40.027457Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:49:50.270277Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:00.636482Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:10.973091Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:21.370808Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:31.700493Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:42.024600Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:52.394112Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T00:50:54.406946Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:50:54.406985Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:50:54.407003Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-01T00:50:54.407009Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T00:50:55.265198Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-01T00:50:55.265224Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-01T00:50:55.265228Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-01T00:50:55.265232Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=26 [2026-11-01T00:50:54.306Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 331 100 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "539c53d4-280b-4cc1-9f85-445cccadb6f2" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "10.132.1.112:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.160:39022 10.134.1.160:443 10.132.0.9:39136 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-01T00:50:55.243Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 21 - "52.71.36.72,10.132.0.9" "python-requests/2.34.2" "2fabf0a9-6697-4a3c-a601-23ef1f04feae" "maas.apps.84f85b540f82405c5762.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.160:443 10.132.0.9:39158 - v1-completions-path