2026-10-30T05:16:44.624682Z info FLAG: --concurrency="0" 2026-10-30T05:16:44.624710Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T05:16:44.624714Z info FLAG: --help="false" 2026-10-30T05:16:44.624716Z info FLAG: --log_as_json="false" 2026-10-30T05:16:44.624718Z info FLAG: --log_caller="" 2026-10-30T05:16:44.624720Z info FLAG: --log_output_level="default:info" 2026-10-30T05:16:44.624722Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T05:16:44.624726Z info FLAG: --log_target="[stdout]" 2026-10-30T05:16:44.624729Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T05:16:44.624731Z info FLAG: --outlierLogPath="" 2026-10-30T05:16:44.624733Z info FLAG: --profiling="true" 2026-10-30T05:16:44.624735Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T05:16:44.624737Z info FLAG: --proxyLogLevel="warning" 2026-10-30T05:16:44.624739Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T05:16:44.624741Z info FLAG: --stsPort="0" 2026-10-30T05:16:44.624743Z info FLAG: --templateFile="" 2026-10-30T05:16:44.624745Z info FLAG: --tokenManagerPlugin="" 2026-10-30T05:16:44.624748Z info FLAG: --vklog="0" 2026-10-30T05:16:44.624750Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T05:16:44.624891Z info Proxy role ips=[10.133.1.8] type=router id=maas-default-gateway-openshift-default-7f694df8bc-h7nkg.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T05:16:44.624938Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T05:16:44.626287Z info cpu limit detected as 2, setting concurrency 2026-10-30T05:16:44.626493Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T05:16:44.626503Z info JWT policy is third-party-jwt 2026-10-30T05:16:44.626507Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T05:16:44.628851Z info Starting default Istio SDS Server 2026-10-30T05:16:44.628886Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T05:16:44.628916Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T05:16:44.629118Z info Opening status port 15020 2026-10-30T05:16:44.629870Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T05:16:44.631267Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T05:16:44.632133Z info Starting proxy agent 2026-10-30T05:16:44.632163Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T05:16:44.641101Z info sds Starting SDS grpc server 2026-10-30T05:16:44.641120Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T05:16:44.696572Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T05:16:44.785744Z info cache generated new workload certificate resourceName=default latency=155.632835ms ttl=23h59m59.21426036s 2026-10-30T05:16:44.785775Z info cache Root cert has changed, start rotating root cert 2026-10-30T05:16:44.785808Z info cache returned workload trust anchor from cache ttl=23h59m59.214192424s 2026-10-30T05:16:44.817461Z info ads ADS: new connection for node:1 2026-10-30T05:16:44.817528Z info cache returned workload certificate from cache ttl=23h59m59.182473453s 2026-10-30T05:16:44.818159Z info ads ADS: new connection for node:2 2026-10-30T05:16:44.818234Z info cache returned workload trust anchor from cache ttl=23h59m59.181767219s 2026-10-30T05:16:44.903169Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:16:53.078575Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:16:54.007853Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-10-30T05:16:54.226220Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-10-30T05:16:55.452216Z info Readiness succeeded in 10.830246191s 2026-10-30T05:16:55.452645Z info Envoy proxy is ready 2026-10-30T05:17:23.814730Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:17:27.221Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 5518 5791 89 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "c9179d33-c248-4b21-8ae4-b0bc900eddbd" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.30:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:43052 10.133.1.8:443 10.132.0.13:43376 - v1-chat-completions-path 2026-10-30T05:17:39.820937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:17:42.414057Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:17:48.717561Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:17:55.998751Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:18:05.773963Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:18:20.552919Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:18:31.458694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:18:33.482210Z info ads ADS: new connection for node:3 2026-10-30T05:18:33.482359Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T05:18:33.482387Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T05:18:35.559121Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:19:15.832943Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:19:16.718Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 342 2087 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "e7126e53-5077-4874-a867-7554143ebbe5" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.1.15:8000" outbound|54321||e2e-prio-pool-14305626-inference-pool-ip-39a4575d.llm.svc.cluster.local 10.133.1.8:54696 10.133.1.8:443 10.132.0.13:34448 - v1-model-routing [2026-10-30T05:19:19.542Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 357 62 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "32e3ed68-c573-43f5-a736-d766d34b2b3c" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.1.15:8000" outbound|54321||e2e-prio-pool-14305626-inference-pool-ip-39a4575d.llm.svc.cluster.local 10.133.1.8:54696 10.133.1.8:443 10.132.0.13:34468 - v1-model-routing [2026-10-30T05:19:20.738Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 347 58 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "90d3aabe-928f-4005-9249-60b5d5d2827f" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.1.15:8000" outbound|54321||e2e-prio-pool-14305626-inference-pool-ip-39a4575d.llm.svc.cluster.local 10.133.1.8:54696 10.133.1.8:443 10.132.0.13:34478 - v1-model-routing 2026-10-30T05:19:34.963963Z info ads ADS: "" 3 terminated with stream closed 2026-10-30T05:19:35.009611Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:19:38.376073Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:19:46.222285Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:21:14.218947Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:21:14.218988Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:21:14.219015Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:21:14.219022Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T05:21:14.137Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 33 325 81 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "ec75ee53-71c1-4a30-b451-8caf458f249d" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.8:52072 10.133.1.8:443 10.132.0.13:34400 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:21:19.999007Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:21:37.175498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:21:57.301069Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:11.572146Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:16.470276Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:20.376009Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:30.683081Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:35.447965Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:50.838682Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:23:21.955231Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 [2026-10-30T05:23:21.892Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 284 78 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "4f782797-c9e2-4dd2-9b6b-ad7a0ccc9e56" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:60310 10.133.1.8:443 10.132.0.13:60238 - v1-completions-path 2026-10-30T05:23:38.386886Z info ads ADS: new connection for node:4 2026-10-30T05:23:38.392121Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T05:23:38.393026Z info cache read certificate from file resource=file-root:system 2026-10-30T05:23:39.455927Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:23:41.534203Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:23:44.464866Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"144","StreamId":"15184491135975025718"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=26 [2026-10-30T05:23:44.199Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 265 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "15276c2b-f9c6-44f1-9875-27447107acc9" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.1.8:443 10.132.0.13:33364 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T05:24:05.750628Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T05:24:05.750661Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T05:24:05.750672Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T05:24:05.750678Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-30T05:24:05.750848Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-10-30T05:24:05.638Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1717 112 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "644ffd3f-9727-4531-bdc4-0580c90a93e0" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.8:46596 10.133.1.8:443 10.132.0.13:54782 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T05:24:57.084023Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:57.084056Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:57.084066Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:57.084071Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T05:24:57.084226Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T05:24:56.977Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1725 107 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "035d9516-ffff-4e4b-b693-1b83d9ae613c" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.8:41432 10.133.1.8:443 10.132.0.13:54654 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T05:25:16.992654Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:25:17.120254Z info ads ADS: "" 4 terminated 2026-10-30T05:25:18.580876Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:25:25.714889Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:25:41.484794Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:25:46.209666Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:26:01.885911Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:26:17.851457Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:26:23.158Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 70 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "8bcd94b6-fe92-4578-94c1-2490319a21c5" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:35142 10.133.1.8:443 10.132.0.13:51336 - v1-completions-path [2026-10-30T05:26:24.483Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 58 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "2237132b-4a9d-4e55-9f25-be6ce898e2bf" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:35142 10.133.1.8:443 10.132.0.13:51344 - v1-completions-path 2026-10-30T05:26:41.463781Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:26:51.764306Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:26:56.446352Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:27:00.164Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 61 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "5df75398-aa1e-415d-9566-c6c66e002cff" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:35142 10.133.1.8:443 10.132.0.13:57664 - v1-completions-path [2026-10-30T05:27:01.350Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 58 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "7061c71a-20b1-4c4b-b58f-6689622fc56a" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:35142 10.133.1.8:443 10.132.0.13:57680 - v1-completions-path 2026-10-30T05:27:13.395167Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:15.906965Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:23.389315Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:33.025205Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:38.430019Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:27:44.023Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 66 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "8ab26c58-ec6b-4b72-84c6-babe7faf1b94" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:60310 10.133.1.8:443 10.132.0.13:52060 - v1-completions-path 2026-10-30T05:27:46.620015Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:51.856577Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:10.967608Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:13.966482Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:29.498679Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:32.373521Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:28:34.529Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 32 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "ccfcb7c5-0949-419e-9303-26083ec817f6" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.8:443 10.132.0.13:52782 - v1-completions-path 2026-10-30T05:28:41.508350Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:47.031792Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T05:28:47.031835Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T05:28:47.031856Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T05:28:47.031863Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-10-30T05:28:46.957Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 309 74 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "82644abe-1223-4e22-ba90-44c292c004de" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.8:49114 10.133.1.8:443 10.132.0.13:48426 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:28:59.523659Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:29:09.901725Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:29:26.565254Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:29:30.074876Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:30:49.049413Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T05:30:49.049483Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T05:30:49.049499Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-30T05:30:49.049504Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-10-30T05:30:48.969Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 80 323 79 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "b006124e-e5df-4373-bce5-39b8bf6b681b" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.8:54530 10.133.1.8:443 10.132.0.13:38022 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T05:30:59.619Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 46 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "2091b4bf-ad66-4d42-b61e-06979b7f3ed9" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.8:443 10.132.0.13:35360 - v1-completions-path 2026-10-30T05:31:01.943698Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:31:05.629169Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:32:10.053752Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T05:32:10.053780Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T05:32:10.053785Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T05:32:10.053790Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-30T05:32:10.008Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 403 - - "-" 73 113 45 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "312b4b88-78c8-4a69-863d-d51479fc792e" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.8:443 10.132.0.13:59918 - v1-completions-path 2026-10-30T05:32:31.620783Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:32:46.765798Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:32:49.338490Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:32:56.749843Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:33:13.668837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:33:16.742009Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:33:32.262457Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:33:43.508741Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:33:48.299820Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:05.313012Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:10.112346Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:13.308435Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:21.171570Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-30T05:34:21.115Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 62 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "4f0c479e-4665-4fdb-8300-cbd535165a0f" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:56844 10.133.1.8:443 10.132.0.13:37488 - v1-completions-path [2026-10-30T05:34:21.849Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 57 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "0c8a8ce6-3b15-4d84-b321-02014deb4658" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:35142 10.133.1.8:443 10.132.0.13:37496 - v1-completions-path [2026-10-30T05:34:23.297Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 58 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "afd2b9a6-69c7-431a-acb5-dd9a5003159c" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:56844 10.133.1.8:443 10.132.0.13:37510 - v1-completions-path [2026-10-30T05:34:23.889Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 57 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "8748f116-9d60-419e-a9b0-1291040c3dd4" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:35142 10.133.1.8:443 10.132.0.13:37520 - v1-completions-path 2026-10-30T05:34:36.333739Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:46.716870Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:51.570452Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:54.598248Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:59.706794Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:35:18.040519Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:35:28.385446Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:35:30.884895Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:35:40.651403Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:35:45.279753Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:36:02.070392Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:36:22.021905Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:36:32.384259Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:36:42.880515Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:36:53.343476Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:03.713085Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:14.104400Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:24.438436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:34.780014Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:45.135761Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:55.520621Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:58.374796Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T05:37:58.374820Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T05:37:58.374823Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T05:37:58.374827Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-30T05:37:58.351Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 23 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "6a80954d-975a-4a41-9a1a-4bf08cbfdaab" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.8:443 10.132.0.13:54992 - v1-completions-path [2026-10-30T05:38:02.936Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 69 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "6633015c-6db6-45ec-890a-4c313cdf5924" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.8:60310 10.133.1.8:443 10.132.0.13:55016 - v1-completions-path