2026-11-03T19:38:45.325440Z info FLAG: --concurrency="0" 2026-11-03T19:38:45.325460Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T19:38:45.325464Z info FLAG: --help="false" 2026-11-03T19:38:45.325466Z info FLAG: --log_as_json="false" 2026-11-03T19:38:45.325468Z info FLAG: --log_caller="" 2026-11-03T19:38:45.325470Z info FLAG: --log_output_level="default:info" 2026-11-03T19:38:45.325472Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T19:38:45.325476Z info FLAG: --log_target="[stdout]" 2026-11-03T19:38:45.325479Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T19:38:45.325481Z info FLAG: --outlierLogPath="" 2026-11-03T19:38:45.325483Z info FLAG: --profiling="true" 2026-11-03T19:38:45.325485Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T19:38:45.325487Z info FLAG: --proxyLogLevel="warning" 2026-11-03T19:38:45.325489Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T19:38:45.325491Z info FLAG: --stsPort="0" 2026-11-03T19:38:45.325493Z info FLAG: --templateFile="" 2026-11-03T19:38:45.325495Z info FLAG: --tokenManagerPlugin="" 2026-11-03T19:38:45.325497Z info FLAG: --vklog="0" 2026-11-03T19:38:45.325500Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T19:38:45.325638Z info Proxy role ips=[10.134.1.12] type=router id=maas-default-gateway-openshift-default-77f556d647-stw9w.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T19:38:45.325682Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T19:38:45.326661Z info cpu limit detected as 2, setting concurrency 2026-11-03T19:38:45.326870Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T19:38:45.326880Z info JWT policy is third-party-jwt 2026-11-03T19:38:45.326883Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T19:38:45.328701Z info Starting default Istio SDS Server 2026-11-03T19:38:45.328735Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T19:38:45.328752Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T19:38:45.329367Z info Opening status port 15020 2026-11-03T19:38:45.329960Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T19:38:45.331308Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T19:38:45.332160Z info Starting proxy agent 2026-11-03T19:38:45.332197Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T19:38:45.341641Z info sds Starting SDS grpc server 2026-11-03T19:38:45.341661Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T19:38:45.398899Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T19:38:45.576177Z info ads ADS: new connection for node:1 2026-11-03T19:38:45.577039Z info ads ADS: new connection for node:2 2026-11-03T19:38:45.621663Z info cache generated new workload certificate resourceName=default latency=291.980516ms ttl=23h59m59.378340975s 2026-11-03T19:38:45.621706Z info cache Root cert has changed, start rotating root cert 2026-11-03T19:38:45.621746Z info cache returned workload trust anchor from cache ttl=23h59m59.378254312s 2026-11-03T19:38:45.621769Z info cache returned workload certificate from cache ttl=23h59m59.378231922s 2026-11-03T19:38:45.621875Z info cache returned workload trust anchor from cache ttl=23h59m59.378127358s 2026-11-03T19:38:45.622219Z info cache returned workload trust anchor from cache ttl=23h59m59.377782686s 2026-11-03T19:38:45.709152Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:38:53.804194Z info Readiness succeeded in 8.48100884s 2026-11-03T19:38:53.804472Z info Envoy proxy is ready 2026-11-03T19:39:31.435386Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T19:39:37.520Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 90 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "569bcf62-c76e-48fc-9944-44beb2900e7b" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "10.132.0.56:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.12:50032 10.134.1.12:443 10.132.0.12:57566 - v1-completions-path 2026-11-03T19:39:53.291295Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:39:57.962266Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:40:24.633034Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:40:29.276750Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:40:34.241193Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:40:42.956229Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T19:40:50.353Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 65 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "582ebc50-21bd-43c7-8f61-c538e200a1c7" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.12:443 10.132.0.12:48618 - v1-completions-path [2026-11-03T19:40:51.159Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 66 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "68cf1667-eb79-4f5a-a489-7a0b75c68e63" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "10.132.0.56:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.12:39192 10.134.1.12:443 10.132.0.12:48628 - v1-completions-path 2026-11-03T19:41:12.466886Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:41:23.239650Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:41:29.688357Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T19:41:40.822Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 288 62 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "35591c3e-b162-4166-9e39-eed65b40188c" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "10.132.0.56:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.12:39192 10.134.1.12:443 10.132.0.12:58234 - v1-completions-path 2026-11-03T19:41:44.644007Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-03T19:41:44.581Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 65 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "54e690e4-8f5b-43ba-81de-896ca2be4061" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "10.132.0.56:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.12:39192 10.134.1.12:443 10.132.0.12:53440 - v1-completions-path 2026-11-03T19:41:58.567035Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:42:03.225689Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:42:13.016399Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T19:42:13.016434Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T19:42:13.016449Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T19:42:13.016454Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T19:42:12.903Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 113 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "357c183f-d533-433a-ae97-2794763b7369" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "10.132.0.53:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.12:41178 10.134.1.12:443 10.132.0.12:60422 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T19:42:13.225Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 74 288 69 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "9b239466-9e7a-44cb-9e49-4c4676e76ae8" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "10.134.0.28:8000" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local 10.134.1.12:50686 10.134.1.12:443 10.132.0.12:60440 - v1-completions-path 2026-11-03T19:42:26.825150Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:42:33.235550Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:42:59.098771Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:43:17.726392Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T19:46:14.431Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 46 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "041123cb-e94d-4a7a-a2dd-3ea157daf45d" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.12:443 10.132.0.12:38394 - v1-completions-path 2026-11-03T19:47:50.190677Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:48:07.664818Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:48:12.293232Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:48:29.197370Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:50:22.514951Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:51:04.228634Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:51:28.613800Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T19:51:28.613846Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T19:51:28.613867Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T19:51:28.613874Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T19:51:28.499Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 80 322 114 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "6537b4cb-2326-48af-89e4-18f5b84c0d88" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "10.132.0.53:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.12:46358 10.134.1.12:443 10.132.0.12:33154 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T19:51:31.027346Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:51:46.465221Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:52:02.397622Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:52:14.891864Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:52:19.772552Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:52:48.681614Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:52:59.508234Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:53:05.905936Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T19:53:05.815Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 90 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "2278256a-f74e-405a-84e3-f5917e76b7a2" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.1.12:47858 10.134.1.12:443 10.132.0.12:38350 - v1-catch-all-path 2026-11-03T19:53:10.680762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:53:17.081159Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T19:53:17.082513Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T19:53:17.013Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 69 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "cdf0ef99-027e-4f76-b144-7800c3b8fbdc" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.1.12:49524 10.134.1.12:443 10.132.0.12:43266 - v1-catch-all-path 2026-11-03T19:53:33.582557Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:53:39.829437Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T19:53:39.829475Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T19:53:39.829490Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T19:53:39.829496Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T19:53:39.708Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 120 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "c2f232cb-e8a7-45ec-945e-c3b09fe9554f" "maas.apps.9a15ddbd156184f738d5.prod.konflux-ocp-ci.dev" "10.133.0.132:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.12:59178 10.134.1.12:443 10.132.0.12:46278 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T19:53:53.785167Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:54:05.051686Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8