2026-10-30T20:06:11.588632Z info FLAG: --concurrency="0" 2026-10-30T20:06:11.588654Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T20:06:11.588658Z info FLAG: --help="false" 2026-10-30T20:06:11.588660Z info FLAG: --log_as_json="false" 2026-10-30T20:06:11.588663Z info FLAG: --log_caller="" 2026-10-30T20:06:11.588665Z info FLAG: --log_output_level="default:info" 2026-10-30T20:06:11.588667Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T20:06:11.588672Z info FLAG: --log_target="[stdout]" 2026-10-30T20:06:11.588674Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T20:06:11.588676Z info FLAG: --outlierLogPath="" 2026-10-30T20:06:11.588678Z info FLAG: --profiling="true" 2026-10-30T20:06:11.588680Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T20:06:11.588683Z info FLAG: --proxyLogLevel="warning" 2026-10-30T20:06:11.588685Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T20:06:11.588688Z info FLAG: --stsPort="0" 2026-10-30T20:06:11.588690Z info FLAG: --templateFile="" 2026-10-30T20:06:11.588692Z info FLAG: --tokenManagerPlugin="" 2026-10-30T20:06:11.588694Z info FLAG: --vklog="0" 2026-10-30T20:06:11.588697Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T20:06:11.588886Z info Proxy role ips=[10.132.0.231] type=router id=maas-default-gateway-openshift-default-6f54779779-bb2r4.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T20:06:11.588943Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T20:06:11.590390Z info cpu limit detected as 2, setting concurrency 2026-10-30T20:06:11.590614Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T20:06:11.590625Z info JWT policy is third-party-jwt 2026-10-30T20:06:11.590629Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T20:06:11.592838Z info Starting default Istio SDS Server 2026-10-30T20:06:11.592864Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T20:06:11.592900Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T20:06:11.593059Z info Opening status port 15020 2026-10-30T20:06:11.594389Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T20:06:11.595995Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T20:06:11.596956Z info Starting proxy agent 2026-10-30T20:06:11.597041Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T20:06:11.597720Z info sds Starting SDS grpc server 2026-10-30T20:06:11.597775Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T20:06:11.686956Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T20:06:11.733876Z info cache generated new workload certificate resourceName=default latency=139.070865ms ttl=23h59m59.266129808s 2026-10-30T20:06:11.733947Z info cache Root cert has changed, start rotating root cert 2026-10-30T20:06:11.733989Z info cache returned workload trust anchor from cache ttl=23h59m59.266012026s 2026-10-30T20:06:11.821246Z info ads ADS: new connection for node:1 2026-10-30T20:06:11.821318Z info cache returned workload certificate from cache ttl=23h59m59.178684462s 2026-10-30T20:06:11.821955Z info ads ADS: new connection for node:2 2026-10-30T20:06:11.822074Z info cache returned workload trust anchor from cache ttl=23h59m59.17792886s 2026-10-30T20:06:11.915070Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:06:20.261743Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:06:21.234200Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-10-30T20:06:22.675868Z info Readiness succeeded in 11.090619396s 2026-10-30T20:06:22.676304Z info Envoy proxy is ready 2026-10-30T20:06:31.623926Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:06:47.143589Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T20:07:04.710Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 87 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "b010699e-0848-491b-8bd4-f0a44b844121" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.231:41694 10.132.0.231:443 10.132.0.13:54028 - v1-completions-path [2026-10-30T20:07:04.986Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "0a99acff-a03e-40a4-9ed2-accf6c9d7eae" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.231:41694 10.132.0.231:443 10.132.0.13:54042 - v1-completions-path 2026-10-30T20:07:20.764983Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:07:31.160452Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:07:35.297230Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:07:40.843162Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-10-30T20:07:40.725Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 118 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "2c01612b-03a4-464c-9d7a-4f6b5d8e1f34" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.231:41694 10.132.0.231:443 10.132.0.13:32792 - v1-completions-path 2026-10-30T20:07:53.758523Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:07:55.991807Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:08:15.194075Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:08:19.838504Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T20:08:34.949Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 75 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "802f6f20-3c35-4fc4-9024-691adc742575" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.231:32792 10.132.0.231:443 10.132.0.13:58908 - v1-completions-path 2026-10-30T20:08:36.369335Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:08:38.490425Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:08:57.473238Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:09:08.296427Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:09:11.329947Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:09:30.442226Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:09:33.003607Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:09:51.766975Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-10-30T20:09:51.767049Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-10-30T20:09:51.767073Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=27 2026-10-30T20:09:51.767081Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 [2026-10-30T20:09:51.686Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 80 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "b785d63b-d34c-4984-bcd1-ea01b854f570" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.44:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.231:36518 10.132.0.231:443 10.132.0.13:45518 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T20:10:04.439717Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:10:06.993477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:11:54.643569Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:11:57.850313Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:13:27.174578Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:13:42.215467Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:13:44.218859Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:13:59.130654Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:16:31.433552Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:17:00.289833Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:17:36.199387Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:17:47.263363Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:17:58.367694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:18:03.419790Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:18:05.352635Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T20:18:05.352763Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T20:18:05.352827Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T20:18:05.352862Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T20:18:05.268Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 84 327 84 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "2db73675-a941-428c-86ea-4d0e01d3e77b" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.44:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.231:50920 10.132.0.231:443 10.132.0.13:58026 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T20:18:06.529764Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:18:17.221576Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:18:21.120703Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:18:37.506635Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:19:15.279924Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T20:19:15.279956Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T20:19:15.279959Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T20:19:15.279963Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T20:19:15.213Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 66 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "a679f768-ae8d-4348-be38-7ded3c1d9229" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.231:443 10.132.0.13:55758 - v1-catch-all-path 2026-10-30T20:19:17.440784Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:19:20.138398Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=27 [2026-10-30T20:19:20.050Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 88 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "241d1b02-2a84-47f0-b8ee-70794b0147b5" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.231:38132 10.132.0.231:443 10.132.0.13:55782 - v1-catch-all-path 2026-10-30T20:19:22.625233Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:19:37.854438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:19:39.883265Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:19:42.574535Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T20:19:42.574590Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T20:19:42.574619Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T20:19:42.574626Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T20:19:40.581Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 1993 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "c75d3bb9-3bb1-4fd0-93b9-11019149a4e7" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.134.1.120:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.231:60914 10.132.0.231:443 10.132.0.13:39102 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T20:19:54.421781Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:20:05.335609Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8