2026-10-31T13:12:58.182989Z info FLAG: --concurrency="0" 2026-10-31T13:12:58.183022Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-31T13:12:58.183028Z info FLAG: --help="false" 2026-10-31T13:12:58.183031Z info FLAG: --log_as_json="false" 2026-10-31T13:12:58.183034Z info FLAG: --log_caller="" 2026-10-31T13:12:58.183037Z info FLAG: --log_output_level="default:info" 2026-10-31T13:12:58.183049Z info FLAG: --log_stacktrace_level="default:none" 2026-10-31T13:12:58.183064Z info FLAG: --log_target="[stdout]" 2026-10-31T13:12:58.183068Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-31T13:12:58.183070Z info FLAG: --outlierLogPath="" 2026-10-31T13:12:58.183073Z info FLAG: --profiling="true" 2026-10-31T13:12:58.183076Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-31T13:12:58.183078Z info FLAG: --proxyLogLevel="warning" 2026-10-31T13:12:58.183081Z info FLAG: --serviceCluster="istio-proxy" 2026-10-31T13:12:58.183084Z info FLAG: --stsPort="0" 2026-10-31T13:12:58.183087Z info FLAG: --templateFile="" 2026-10-31T13:12:58.183090Z info FLAG: --tokenManagerPlugin="" 2026-10-31T13:12:58.183094Z info FLAG: --vklog="0" 2026-10-31T13:12:58.183098Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-31T13:12:58.183335Z info Proxy role ips=[10.132.0.89] type=router id=maas-default-gateway-openshift-default-7f5667f86c-ckhwx.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-31T13:12:58.183454Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-31T13:12:58.189760Z info cpu limit detected as 2, setting concurrency 2026-10-31T13:12:58.190163Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-31T13:12:58.190181Z info JWT policy is third-party-jwt 2026-10-31T13:12:58.190186Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-31T13:12:58.193625Z info Starting default Istio SDS Server 2026-10-31T13:12:58.193652Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-31T13:12:58.193677Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-31T13:12:58.194312Z info Opening status port 15020 2026-10-31T13:12:58.207436Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-31T13:12:58.209733Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-31T13:12:58.211176Z info Starting proxy agent 2026-10-31T13:12:58.211270Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-31T13:12:58.213278Z info sds Starting SDS grpc server 2026-10-31T13:12:58.213324Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-31T13:12:58.379022Z info cache generated new workload certificate resourceName=default latency=170.798574ms ttl=23h59m59.62098285s 2026-10-31T13:12:58.379089Z info cache Root cert has changed, start rotating root cert 2026-10-31T13:12:58.379155Z info cache returned workload trust anchor from cache ttl=23h59m59.6208467s 2026-10-31T13:12:58.489959Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-31T13:12:58.839585Z info ads ADS: new connection for node:1 2026-10-31T13:12:58.844257Z info cache returned workload certificate from cache ttl=23h59m59.155747593s 2026-10-31T13:12:58.854264Z info ads ADS: new connection for node:2 2026-10-31T13:12:58.854832Z info cache returned workload trust anchor from cache ttl=23h59m59.145172788s 2026-10-31T13:12:59.039447Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:13:00.743657Z error failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused 2026-10-31T13:13:07.677205Z info Readiness succeeded in 9.505607614s 2026-10-31T13:13:07.677215Z info Readiness succeeded in 9.505627301s 2026-10-31T13:13:07.677703Z info Envoy proxy is ready 2026-10-31T13:16:30.802989Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:16:53.078601Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:17:37.373426Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:17:42.842050Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T13:17:42.842091Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T13:17:42.842106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-31T13:17:42.842111Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-10-31T13:17:42.762Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 87 329 79 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "7842c8f9-27f5-4bab-a55d-e1c33254d42c" "maas.apps.57b5c3f50138fe150efa.prod.konflux-ocp-ci.dev" "10.133.0.26:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.89:57782 10.132.0.89:443 10.132.0.10:50408 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T13:17:55.663979Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:17:57.594769Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T13:18:04.628Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 74 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "ff6396c1-e378-4f0c-8560-752e16c3f14b" "maas.apps.57b5c3f50138fe150efa.prod.konflux-ocp-ci.dev" "10.133.0.30:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.89:39638 10.132.0.89:443 10.132.0.10:59296 - v1-completions-path 2026-10-31T13:18:16.814284Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:18:18.649549Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:18:21.505270Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:18:29.732789Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T13:18:37.484Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 75 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "1244f454-fe19-4e0d-b175-8d69cb59def5" "maas.apps.57b5c3f50138fe150efa.prod.konflux-ocp-ci.dev" "10.133.0.30:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.89:44068 10.132.0.89:443 10.132.0.10:50298 - v1-completions-path [2026-10-31T13:18:38.574Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 54 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "7b82f50a-e364-4a55-b3f2-79633b1666d4" "maas.apps.57b5c3f50138fe150efa.prod.konflux-ocp-ci.dev" "10.133.0.30:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.89:44068 10.132.0.89:443 10.132.0.10:50318 - v1-completions-path [2026-10-31T13:18:39.323Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 299 62 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "8497a075-3f42-409e-a771-b48a781e3915" "maas.apps.57b5c3f50138fe150efa.prod.konflux-ocp-ci.dev" "10.133.0.30:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.89:39638 10.132.0.89:443 10.132.0.10:50328 - v1-completions-path 2026-10-31T13:18:54.328501Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:19:04.561555Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:19:06.492727Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:19:11.894834Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T13:19:11.894873Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T13:19:11.894889Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T13:19:11.894894Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-31T13:19:11.818Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 317 76 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "44360777-656c-445e-8141-cdb44bcaa787" "maas.apps.57b5c3f50138fe150efa.prod.konflux-ocp-ci.dev" "10.133.0.26:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.89:60792 10.132.0.89:443 10.132.0.10:53340 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T13:19:25.274122Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:19:27.187504Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:19:43.654000Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:19:47.754436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:20:04.514943Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:20:06.303615Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:20:22.268088Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:20:32.968850Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:20:35.596209Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:20:54.936525Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:20:57.223192Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:21:08.013868Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:21:26.278037Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:21:36.534447Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:21:52.933654Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:21:55.758423Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T13:23:36.755Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 47 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "7022854a-6992-43b9-9f02-44b2945fef5d" "maas.apps.57b5c3f50138fe150efa.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.89:443 10.132.0.10:39880 - v1-completions-path 2026-10-31T13:23:38.429482Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:23:41.363973Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:25:08.899405Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:25:23.961094Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:25:25.788946Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:25:31.131303Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 [2026-10-31T13:25:31.073Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 283 77 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "df260a32-d332-4c9a-acdf-94cd12adb9b1" "maas.apps.57b5c3f50138fe150efa.prod.konflux-ocp-ci.dev" "10.132.0.57:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.89:53182 10.132.0.89:443 10.132.0.10:60628 - v1-completions-path 2026-10-31T13:25:43.065424Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:27:06.221871Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:27:28.580350Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:27:42.174679Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T13:27:42.174735Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T13:27:42.174764Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-31T13:27:42.174771Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-10-31T13:27:42.113Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 628 61 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "7a9a9449-1515-42f3-98ec-19335046fba2" "maas.apps.57b5c3f50138fe150efa.prod.konflux-ocp-ci.dev" "10.133.0.26:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.89:55940 10.132.0.89:443 10.132.0.10:60102 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T13:27:43.576102Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:27:54.573386Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:28:05.662012Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:28:08.676166Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:28:19.905491Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:28:22.218777Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T13:28:28.529Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 58 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "041dfbde-ed8f-4869-a650-56b4e7d23bd4" "maas.apps.57b5c3f50138fe150efa.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.89:443 10.132.0.10:40162 - v1-completions-path 2026-10-31T13:28:40.330718Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:31:31.783651Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:31:37.013310Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:31:52.297551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:31:54.125925Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:32:08.497846Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T13:32:19.139916Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8