2026-11-03T21:57:04.606645Z info FLAG: --concurrency="0" 2026-11-03T21:57:04.606663Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T21:57:04.606667Z info FLAG: --help="false" 2026-11-03T21:57:04.606669Z info FLAG: --log_as_json="false" 2026-11-03T21:57:04.606671Z info FLAG: --log_caller="" 2026-11-03T21:57:04.606674Z info FLAG: --log_output_level="default:info" 2026-11-03T21:57:04.606676Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T21:57:04.606680Z info FLAG: --log_target="[stdout]" 2026-11-03T21:57:04.606683Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T21:57:04.606684Z info FLAG: --outlierLogPath="" 2026-11-03T21:57:04.606686Z info FLAG: --profiling="true" 2026-11-03T21:57:04.606688Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T21:57:04.606691Z info FLAG: --proxyLogLevel="warning" 2026-11-03T21:57:04.606692Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T21:57:04.606694Z info FLAG: --stsPort="0" 2026-11-03T21:57:04.606696Z info FLAG: --templateFile="" 2026-11-03T21:57:04.606699Z info FLAG: --tokenManagerPlugin="" 2026-11-03T21:57:04.606701Z info FLAG: --vklog="0" 2026-11-03T21:57:04.606704Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T21:57:04.606849Z info Proxy role ips=[10.133.0.26] type=router id=maas-default-gateway-openshift-default-66db998887-6czqx.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T21:57:04.606897Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T21:57:04.608236Z info cpu limit detected as 2, setting concurrency 2026-11-03T21:57:04.608430Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T21:57:04.608440Z info JWT policy is third-party-jwt 2026-11-03T21:57:04.608443Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T21:57:04.610740Z info Starting default Istio SDS Server 2026-11-03T21:57:04.610767Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T21:57:04.610786Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T21:57:04.610950Z info Opening status port 15020 2026-11-03T21:57:04.611613Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T21:57:04.611784Z info sds Starting SDS grpc server 2026-11-03T21:57:04.611798Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T21:57:04.612976Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T21:57:04.613772Z info Starting proxy agent 2026-11-03T21:57:04.613803Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T21:57:04.677450Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T21:57:04.706894Z info cache generated new workload certificate resourceName=default latency=95.016297ms ttl=23h59m59.293111745s 2026-11-03T21:57:04.706943Z info cache Root cert has changed, start rotating root cert 2026-11-03T21:57:04.706982Z info cache returned workload trust anchor from cache ttl=23h59m59.293019564s 2026-11-03T21:57:04.777140Z info ads ADS: new connection for node:1 2026-11-03T21:57:04.777217Z info cache returned workload certificate from cache ttl=23h59m59.222786051s 2026-11-03T21:57:04.777376Z info ads ADS: new connection for node:2 2026-11-03T21:57:04.777452Z info cache returned workload trust anchor from cache ttl=23h59m59.222548823s 2026-11-03T21:57:04.858637Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:57:12.746946Z info Readiness succeeded in 8.142526028s 2026-11-03T21:57:12.747341Z info Envoy proxy is ready 2026-11-03T21:59:06.940174Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T21:59:06.940231Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T21:59:06.940256Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T21:59:06.940263Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T21:59:06.826Z] "DELETE /v1/api-keys/8e5c1c82-e950-4d35-ae11-d21a9e61b8ba HTTP/1.1" 200 - via_upstream - "-" 0 370 113 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "8198db0b-b640-4925-8d47-d2e9064d59a8" "maas.apps.6a40f9c86944f74aa03d.prod.konflux-ocp-ci.dev" "10.132.0.57:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.26:56528 10.133.0.26:443 10.132.0.12:58948 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T21:59:14.270032Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T21:59:20.069Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 86 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "c94ac594-f11d-4b32-a536-30c393fb54c3" "maas.apps.6a40f9c86944f74aa03d.prod.konflux-ocp-ci.dev" "10.132.0.61:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.26:33170 10.133.0.26:443 10.132.0.12:33558 - v1-completions-path [2026-11-03T21:59:20.211Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 283 67 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "4a73c91d-7413-4c28-8852-8e23bf86917b" "maas.apps.6a40f9c86944f74aa03d.prod.konflux-ocp-ci.dev" "10.132.0.61:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.26:33182 10.133.0.26:443 10.132.0.12:33562 - v1-completions-path 2026-11-03T21:59:28.061573Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:00:26.183615Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:00:48.765250Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:01:29.690269Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:01:45.509837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:01:47.015771Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:02:04.156834Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:02:05.700103Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:02:08.433904Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:02:16.863700Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:02:22.936835Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T22:02:22.936871Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T22:02:22.936886Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T22:02:22.936891Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T22:02:22.873Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 77 319 63 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "21da2c1d-a5d3-465d-85cc-ca55e865ca77" "maas.apps.6a40f9c86944f74aa03d.prod.konflux-ocp-ci.dev" "10.132.0.57:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.26:51000 10.133.0.26:443 10.132.0.12:41672 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T22:02:41.065347Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:02:51.380559Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:02:53.037079Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T22:03:00.172Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 73 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "28443bab-4790-408a-9628-0873edd6e018" "maas.apps.6a40f9c86944f74aa03d.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.26:54096 10.133.0.26:443 10.132.0.12:39990 - v1-completions-path 2026-11-03T22:03:11.998481Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:03:13.630843Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:03:18.753915Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-03T22:03:18.695Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 74 297 75 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "b258fa31-72c6-498e-abbc-be0c836fb023" "maas.apps.6a40f9c86944f74aa03d.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local 10.133.0.26:45106 10.133.0.26:443 10.132.0.12:60284 - v1-completions-path 2026-11-03T22:03:30.128067Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:03:32.151380Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:03:48.919506Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:03:50.351017Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:04:06.703496Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:04:17.414960Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:04:20.090036Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:04:35.480246Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:04:36.927653Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T22:04:43.318Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 0 34 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "52c476bd-9071-4a95-80f1-b93d1d97b8a7" "maas.apps.6a40f9c86944f74aa03d.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.26:443 10.132.0.12:59520 - v1-completions-path 2026-11-03T22:04:56.009786Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:04:57.593915Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:06:26.035951Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T22:06:25.022Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 48 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "6171ea86-519b-4625-8166-f5748fa98477" "maas.apps.6a40f9c86944f74aa03d.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.26:443 10.132.0.12:43628 - v1-completions-path 2026-11-03T22:06:27.960655Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:07:53.716978Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:08:08.550625Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:08:10.042665Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:08:13.111483Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T22:08:13.111511Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T22:08:13.111514Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T22:08:13.111518Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T22:08:13.076Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 34 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "768049d7-2c06-4a01-af6d-8752e77fdeb1" "maas.apps.6a40f9c86944f74aa03d.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.26:443 10.132.0.12:55372 - v1-completions-path 2026-11-03T22:08:24.505046Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:09:37.629946Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:09:54.139388Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:10:09.106243Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T22:10:09.106278Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T22:10:09.106294Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T22:10:09.106299Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T22:10:09.039Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 80 323 66 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "7b888153-4a22-47e1-9353-08018add09bd" "maas.apps.6a40f9c86944f74aa03d.prod.konflux-ocp-ci.dev" "10.132.0.57:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.26:49864 10.133.0.26:443 10.132.0.12:47700 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T22:10:10.342517Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:10:22.030905Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:10:25.693197Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:10:37.033422Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:10:39.563103Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:10:57.414730Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:11:01.700663Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:11:06.838502Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:11:22.111257Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:11:23.529218Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:11:38.105176Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T22:11:48.867193Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9