2026-11-03T19:27:40.301926Z info FLAG: --concurrency="0" 2026-11-03T19:27:40.301945Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T19:27:40.301949Z info FLAG: --help="false" 2026-11-03T19:27:40.301952Z info FLAG: --log_as_json="false" 2026-11-03T19:27:40.301954Z info FLAG: --log_caller="" 2026-11-03T19:27:40.301956Z info FLAG: --log_output_level="default:info" 2026-11-03T19:27:40.301958Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T19:27:40.301962Z info FLAG: --log_target="[stdout]" 2026-11-03T19:27:40.301965Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T19:27:40.301967Z info FLAG: --outlierLogPath="" 2026-11-03T19:27:40.301969Z info FLAG: --profiling="true" 2026-11-03T19:27:40.301971Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T19:27:40.301973Z info FLAG: --proxyLogLevel="warning" 2026-11-03T19:27:40.301975Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T19:27:40.301977Z info FLAG: --stsPort="0" 2026-11-03T19:27:40.301979Z info FLAG: --templateFile="" 2026-11-03T19:27:40.301981Z info FLAG: --tokenManagerPlugin="" 2026-11-03T19:27:40.301983Z info FLAG: --vklog="0" 2026-11-03T19:27:40.301986Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T19:27:40.302134Z info Proxy role ips=[10.134.0.135] type=router id=maas-default-gateway-openshift-default-7854949c9-q7v6d.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T19:27:40.302213Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T19:27:40.303432Z info cpu limit detected as 2, setting concurrency 2026-11-03T19:27:40.303650Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T19:27:40.303662Z info JWT policy is third-party-jwt 2026-11-03T19:27:40.303665Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T19:27:40.305939Z info Starting default Istio SDS Server 2026-11-03T19:27:40.305965Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T19:27:40.305989Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T19:27:40.306312Z info Opening status port 15020 2026-11-03T19:27:40.307602Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T19:27:40.308921Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T19:27:40.309802Z info Starting proxy agent 2026-11-03T19:27:40.309839Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T19:27:40.317645Z info sds Starting SDS grpc server 2026-11-03T19:27:40.317670Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T19:27:40.377763Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T19:27:40.546789Z info ads ADS: new connection for node:1 2026-11-03T19:27:40.547244Z info ads ADS: new connection for node:2 2026-11-03T19:27:40.615829Z info cache generated new workload certificate resourceName=default latency=307.909408ms ttl=23h59m59.384175416s 2026-11-03T19:27:40.615867Z info cache Root cert has changed, start rotating root cert 2026-11-03T19:27:40.615904Z info cache returned workload certificate from cache ttl=23h59m59.384101415s 2026-11-03T19:27:40.615914Z info cache returned workload trust anchor from cache ttl=23h59m59.384086047s 2026-11-03T19:27:40.616014Z info cache returned workload trust anchor from cache ttl=23h59m59.383986359s 2026-11-03T19:27:40.616332Z info cache returned workload trust anchor from cache ttl=23h59m59.383668154s 2026-11-03T19:27:40.713994Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:27:46.346766Z error failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused 2026-11-03T19:27:48.854880Z info Readiness succeeded in 8.555892975s 2026-11-03T19:27:48.855285Z info Envoy proxy is ready 2026-11-03T19:27:49.240646Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:28:45.125824Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T19:28:50.946Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 85 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "86ad7329-ab0e-43f9-8fdd-fa86fe78055e" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.135:41144 10.134.0.135:443 10.132.0.22:34012 - v1-completions-path [2026-11-03T19:28:51.342Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 64 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "6a4d2020-7467-40fd-a6ab-6f789ee73a3c" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.135:41152 10.134.0.135:443 10.132.0.22:34028 - v1-completions-path [2026-11-03T19:28:52.082Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 61 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "4c93543c-3ac8-4beb-ad58-372f6a00c8d5" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.135:41152 10.134.0.135:443 10.132.0.22:34034 - v1-completions-path 2026-11-03T19:29:06.516429Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:29:11.345373Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:29:37.936596Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:29:56.577598Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:30:04.821233Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T19:30:04.821272Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T19:30:04.821287Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T19:30:04.821293Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T19:30:04.704Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 77 320 116 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "e6178045-9cb6-4d61-9535-dc7b142275be" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.133.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.135:40596 10.134.0.135:443 10.132.0.22:39576 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T19:30:05.164Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 299 61 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "34d9fb86-388a-4935-bcb3-f1c62fd4bbb5" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.135:41152 10.134.0.135:443 10.132.0.22:39578 - v1-completions-path 2026-11-03T19:30:07.681793Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-03T19:30:07.676Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 59 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "3f8c86e4-1bc0-4de5-8916-5661c8d53556" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.135:41152 10.134.0.135:443 10.132.0.22:52130 - v1-completions-path [2026-11-03T19:30:10.036Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 55 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "876cd044-b0bc-42f2-aeb3-7aab1878692f" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.135:41152 10.134.0.135:443 10.132.0.22:52136 - v1-completions-path [2026-11-03T19:30:10.546Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 57 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "78fda2fe-b808-4fce-a276-44dee4b9f242" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.135:41152 10.134.0.135:443 10.132.0.22:52148 - v1-completions-path [2026-11-03T19:30:10.789Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 56 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "c563918d-2bb2-4d3c-b1d8-7bc3c61ae68d" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.135:41152 10.134.0.135:443 10.132.0.22:52156 - v1-completions-path 2026-11-03T19:30:28.562922Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:30:39.678427Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:30:44.426721Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T19:31:00.117Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 59 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "47312b12-1297-422c-b8fc-447bf90b499b" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.135:41144 10.134.0.135:443 10.132.0.22:51048 - v1-completions-path 2026-11-03T19:31:14.029766Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:31:18.806602Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:31:45.167459Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:31:51.663277Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:32:17.660914Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:32:39.526563Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:32:52.644851Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:32:58.963933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:33:26.391173Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:33:31.947345Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:33:51.870280Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:34:14.849098Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:34:25.846190Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:34:46.400745Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:34:52.629949Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:37:15.499943Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:37:21.863869Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:39:00.236077Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:39:17.419467Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:39:22.135746Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:39:46.505740Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:41:51.778849Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:42:35.808058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:43:05.841963Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:43:21.414059Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:43:28.416322Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:43:39.151848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:43:44.075175Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:44:07.831881Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:47:44.196204Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T19:47:44.196236Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T19:47:44.196239Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T19:47:44.196242Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T19:47:43.987Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 208 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "74c9a2d7-b445-4c04-b333-fa94b9038f26" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.135:443 10.132.0.22:39206 - v1-catch-all-path 2026-11-03T19:47:53.284594Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:48:06.244903Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:48:12.402360Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T19:48:12.419059Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T19:48:12.340Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 78 - "44.212.242.249,10.132.0.22" "python-requests/2.34.2" "a59f16d2-3579-48c6-bba5-fec35772b4a9" "maas.apps.5f1bc85df3ec58d572e7.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.135:53568 10.134.0.135:443 10.132.0.22:50176 - v1-catch-all-path 2026-11-03T19:48:27.090291Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:48:31.793164Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:48:48.499095Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T19:49:00.192147Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8