2026-10-30T22:34:07.682821Z info FLAG: --concurrency="0" 2026-10-30T22:34:07.682841Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T22:34:07.682845Z info FLAG: --help="false" 2026-10-30T22:34:07.682847Z info FLAG: --log_as_json="false" 2026-10-30T22:34:07.682849Z info FLAG: --log_caller="" 2026-10-30T22:34:07.682851Z info FLAG: --log_output_level="default:info" 2026-10-30T22:34:07.682853Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T22:34:07.682861Z info FLAG: --log_target="[stdout]" 2026-10-30T22:34:07.682863Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T22:34:07.682865Z info FLAG: --outlierLogPath="" 2026-10-30T22:34:07.682867Z info FLAG: --profiling="true" 2026-10-30T22:34:07.682869Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T22:34:07.682871Z info FLAG: --proxyLogLevel="warning" 2026-10-30T22:34:07.682873Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T22:34:07.682875Z info FLAG: --stsPort="0" 2026-10-30T22:34:07.682877Z info FLAG: --templateFile="" 2026-10-30T22:34:07.682879Z info FLAG: --tokenManagerPlugin="" 2026-10-30T22:34:07.682885Z info FLAG: --vklog="0" 2026-10-30T22:34:07.682888Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T22:34:07.683045Z info Proxy role ips=[10.134.0.120] type=router id=maas-default-gateway-openshift-default-694db55c87-5mdln.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T22:34:07.683090Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T22:34:07.684265Z info cpu limit detected as 2, setting concurrency 2026-10-30T22:34:07.684456Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T22:34:07.684467Z info JWT policy is third-party-jwt 2026-10-30T22:34:07.684470Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T22:34:07.686419Z info Starting default Istio SDS Server 2026-10-30T22:34:07.686438Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T22:34:07.686456Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T22:34:07.686790Z info Opening status port 15020 2026-10-30T22:34:07.687526Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T22:34:07.688873Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T22:34:07.689770Z info Starting proxy agent 2026-10-30T22:34:07.689803Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T22:34:07.698956Z info sds Starting SDS grpc server 2026-10-30T22:34:07.698978Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T22:34:07.754843Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T22:34:07.794168Z info cache generated new workload certificate resourceName=default latency=106.747147ms ttl=23h59m59.205836868s 2026-10-30T22:34:07.794211Z info cache Root cert has changed, start rotating root cert 2026-10-30T22:34:07.794246Z info cache returned workload trust anchor from cache ttl=23h59m59.205754707s 2026-10-30T22:34:07.902849Z info ads ADS: new connection for node:1 2026-10-30T22:34:07.902954Z info cache returned workload certificate from cache ttl=23h59m59.09704796s 2026-10-30T22:34:07.903848Z info ads ADS: new connection for node:2 2026-10-30T22:34:07.903956Z info cache returned workload trust anchor from cache ttl=23h59m59.096075911s 2026-10-30T22:34:07.986098Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:34:16.355004Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:34:17.300203Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-10-30T22:34:18.164041Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-10-30T22:34:18.343738Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:34:19.273512Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-10-30T22:34:20.224004Z info Readiness succeeded in 12.543467539s 2026-10-30T22:34:20.224324Z info Envoy proxy is ready 2026-10-30T22:34:43.407633Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:34:56.500125Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:35:09.298Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 91 - "34.228.250.194,10.132.0.16" "python-requests/2.34.2" "42752640-60c5-4a3f-9e77-94ed43ceb689" "maas.apps.184a73d47cc96ae9e775.prod.konflux-ocp-ci.dev" "10.133.0.25:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.120:45496 10.134.0.120:443 10.132.0.16:40200 - v1-completions-path 2026-10-30T22:35:27.548162Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:35:38.085887Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:35:44.641800Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:35:57.841Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 60 - "34.228.250.194,10.132.0.16" "python-requests/2.34.2" "d4490cc2-665c-40b5-abc9-7f1599ebdaa1" "maas.apps.184a73d47cc96ae9e775.prod.konflux-ocp-ci.dev" "10.133.0.25:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.120:45496 10.134.0.120:443 10.132.0.16:59254 - v1-completions-path 2026-10-30T22:36:13.555594Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:36:25.590955Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:36:45.283676Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:36:51.535364Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:37:17.654299Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:37:39.623292Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:37:50.976714Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:38:02.194014Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:38:08.517Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 78 - "34.228.250.194,10.132.0.16" "python-requests/2.34.2" "630e1b1c-b228-4ba2-90f9-8ff9704e308a" "maas.apps.184a73d47cc96ae9e775.prod.konflux-ocp-ci.dev" "10.133.0.24:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.120:40434 10.134.0.120:443 10.132.0.16:44502 - v1-completions-path 2026-10-30T22:39:02.307830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:39:44.071786Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:39:54.650058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:44:16.982653Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-30T22:44:16.993559Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:44:16.993595Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:44:16.993615Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T22:44:16.993620Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T22:44:16.877Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 116 - "34.228.250.194,10.132.0.16" "python-requests/2.34.2" "02a43811-9cea-499b-8ffb-0c4b266f0188" "maas.apps.184a73d47cc96ae9e775.prod.konflux-ocp-ci.dev" "10.134.0.48:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.120:33270 10.134.0.120:443 10.132.0.16:58840 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T22:44:40.014127Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:44:52.226Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 73 285 86 - "34.228.250.194,10.132.0.16" "python-requests/2.34.2" "fdfd9eaa-f8f7-4b0a-9231-6cf406a90e17" "maas.apps.184a73d47cc96ae9e775.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.120:43664 10.134.0.120:443 10.132.0.16:56428 - v1-completions-path 2026-10-30T22:45:06.753861Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:45:11.271985Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:45:35.772100Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:47:40.845504Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:48:23.664986Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:48:51.786317Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:48:51.786359Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:48:51.786378Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:48:51.786383Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T22:48:51.716Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 628 69 - "34.228.250.194,10.132.0.16" "python-requests/2.34.2" "2f4917eb-df97-4085-9e5b-a7334d6a9f22" "maas.apps.184a73d47cc96ae9e775.prod.konflux-ocp-ci.dev" "10.134.0.48:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.120:43372 10.134.0.120:443 10.132.0.16:32808 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T22:48:53.754006Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:49:09.242352Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:49:17.668983Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:49:29.171602Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:49:34.048623Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:49:41.611527Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:49:41.611570Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:49:41.611587Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T22:49:41.611592Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T22:49:41.555Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 524 56 - "34.228.250.194,10.132.0.16" "python-requests/2.34.2" "2032dbda-a54d-4621-8b89-51e0c4cced1e" "maas.apps.184a73d47cc96ae9e775.prod.konflux-ocp-ci.dev" "10.134.0.48:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.120:60794 10.134.0.120:443 10.132.0.16:47614 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T22:49:55.245217Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:53:23.547860Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T22:53:23.547886Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T22:53:23.547889Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T22:53:23.547893Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-30T22:53:23.384Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 162 - "34.228.250.194,10.132.0.16" "python-requests/2.34.2" "1da9024f-8203-409d-9be8-cbbe406192b0" "maas.apps.184a73d47cc96ae9e775.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.120:443 10.132.0.16:60954 - v1-catch-all-path 2026-10-30T22:53:30.958402Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:53:42.430028Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:53:49.617331Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-10-30T22:53:49.542Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 75 - "34.228.250.194,10.132.0.16" "python-requests/2.34.2" "9f9e1b88-2a85-4cf4-a8dc-84bacf578dd3" "maas.apps.184a73d47cc96ae9e775.prod.konflux-ocp-ci.dev" "10.133.0.28:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.120:45212 10.134.0.120:443 10.132.0.16:40192 - v1-catch-all-path 2026-10-30T22:54:03.483275Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:54:08.153136Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:54:24.499130Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:54:36.014789Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8