2026-11-02T22:07:26.519337Z info FLAG: --concurrency="0" 2026-11-02T22:07:26.519359Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T22:07:26.519364Z info FLAG: --help="false" 2026-11-02T22:07:26.519367Z info FLAG: --log_as_json="false" 2026-11-02T22:07:26.519370Z info FLAG: --log_caller="" 2026-11-02T22:07:26.519373Z info FLAG: --log_output_level="default:info" 2026-11-02T22:07:26.519376Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T22:07:26.519382Z info FLAG: --log_target="[stdout]" 2026-11-02T22:07:26.519386Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T22:07:26.519389Z info FLAG: --outlierLogPath="" 2026-11-02T22:07:26.519392Z info FLAG: --profiling="true" 2026-11-02T22:07:26.519395Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T22:07:26.519398Z info FLAG: --proxyLogLevel="warning" 2026-11-02T22:07:26.519402Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T22:07:26.519405Z info FLAG: --stsPort="0" 2026-11-02T22:07:26.519407Z info FLAG: --templateFile="" 2026-11-02T22:07:26.519410Z info FLAG: --tokenManagerPlugin="" 2026-11-02T22:07:26.519413Z info FLAG: --vklog="0" 2026-11-02T22:07:26.519417Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T22:07:26.519636Z info Proxy role ips=[10.133.0.36] type=router id=maas-default-gateway-openshift-default-696f47c6c-gjn6k.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T22:07:26.519704Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T22:07:26.520943Z info cpu limit detected as 2, setting concurrency 2026-11-02T22:07:26.521168Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T22:07:26.521182Z info JWT policy is third-party-jwt 2026-11-02T22:07:26.521186Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T22:07:26.522932Z info Starting default Istio SDS Server 2026-11-02T22:07:26.522959Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T22:07:26.522984Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T22:07:26.523368Z info Opening status port 15020 2026-11-02T22:07:26.524319Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T22:07:26.525660Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T22:07:26.526563Z info Starting proxy agent 2026-11-02T22:07:26.526592Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T22:07:26.539712Z info sds Starting SDS grpc server 2026-11-02T22:07:26.539746Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T22:07:26.584880Z info cache generated new workload certificate resourceName=default latency=60.747203ms ttl=23h59m59.415123536s 2026-11-02T22:07:26.584913Z info cache Root cert has changed, start rotating root cert 2026-11-02T22:07:26.584948Z info cache returned workload trust anchor from cache ttl=23h59m59.415052549s 2026-11-02T22:07:26.608891Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T22:07:26.718466Z info ads ADS: new connection for node:1 2026-11-02T22:07:26.718576Z info cache returned workload certificate from cache ttl=23h59m59.281426228s 2026-11-02T22:07:26.718815Z info ads ADS: new connection for node:2 2026-11-02T22:07:26.718886Z info cache returned workload trust anchor from cache ttl=23h59m59.28111517s 2026-11-02T22:07:26.814417Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:07:34.911664Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:07:35.846729Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-02T22:07:36.462301Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-02T22:07:37.376026Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:07:37.466877Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:07:37.490981Z info Readiness succeeded in 10.974348035s 2026-11-02T22:07:37.491413Z info Envoy proxy is ready [2026-11-02T22:08:11.970Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 384 86 - "44.212.242.249,10.133.0.11" "curl/7.76.1" "0ab9623e-1d62-4b94-8566-6bb13cb1b085" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:51276 10.133.0.36:443 10.133.0.11:41230 - v1-model-routing [2026-11-02T22:08:12.130Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 366 62 - "44.212.242.249,10.133.0.11" "curl/7.76.1" "2280353a-9c2e-46c6-b6c5-27248e0a6aba" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:51282 10.133.0.36:443 10.133.0.11:41242 - v1-model-routing 2026-11-02T22:08:13.680073Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:08:13.680106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:08:13.680124Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:08:13.680130Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T22:08:13.667Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 12 - "44.212.242.249,10.133.0.11" "curl/7.76.1" "540b5efa-e3ef-4a2f-aabe-6ee5945f1a4d" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.36:37580 10.133.0.36:443 10.133.0.11:41270 - odh-ai-gateway-infra.maas-api-route.4 [2026-11-02T22:10:20.228Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 40 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "be2388c8-86a8-434c-bb5e-1a4d040ae57f" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.36:443 10.133.0.11:51196 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:10:20.865328Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:20.865361Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:20.865385Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T22:10:20.865390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T22:10:20.781Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 84 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "12eea2e5-162f-4496-8f54-e64ecc59fbcc" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.36:38326 10.133.0.36:443 10.133.0.11:51216 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T22:10:21.084Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 99 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "ed77d1ce-edc5-48f1-94a1-01ffe07994eb" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.36:443 10.133.0.11:55358 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:10:21.493226Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:21.493260Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:21.493274Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T22:10:21.493279Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:22.110387Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:22.110421Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:22.110436Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T22:10:22.110440Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T22:10:21.408Z] "DELETE /v1/api-keys/7ea322aa-5a23-48f7-be89-12813eed0ba0 HTTP/1.1" 200 - via_upstream - "-" 0 321 84 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "6cd69d5e-7467-4100-80b8-50756ecca11c" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.36:38326 10.133.0.36:443 10.133.0.11:55360 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T22:10:22.019Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 93 316 90 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "eba5e451-ba1c-48ce-8e04-d73b67f2eb02" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.36:38326 10.133.0.36:443 10.133.0.11:55374 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:10:27.582993Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T22:10:27.583018Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T22:10:27.583021Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T22:10:27.583025Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-02T22:10:27.549Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 77 33 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "aa946530-f0b4-48a1-8498-7e352a00ffa7" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.36:443 10.133.0.11:55396 - v1-completions-path 2026-11-02T22:10:29.251181Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:10:29.251234Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:10:29.251256Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:10:29.251269Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T22:10:29.193Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 57 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "22c163c0-821c-48a9-96b4-bdf3ba5c888e" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.36:38330 10.133.0.36:443 10.133.0.11:55408 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T22:10:29.440Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 66 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "0d395877-ae6e-4a85-83f6-2dc4f663116a" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:51276 10.133.0.36:443 10.133.0.11:55420 - v1-completions-path 2026-11-02T22:10:37.042359Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:10:51.863708Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:14:16.540046Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:15:23.400822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:16:34.684693Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:17:16.212670Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:17:54.383277Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:18:37.805628Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:19:52.169141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:20:03.563227Z warn xdsproxy upstream terminated with unexpected error rpc error: code = Unavailable desc = error reading from server: read tcp 10.133.0.36:60132->172.31.106.197:15012: read: connection reset by peer id=1 2026-11-02T22:20:04.146556Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-02T22:20:22.690587Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:26:31.678903Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=4 2026-11-02T22:27:20.210267Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:27:45.969096Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:28:32.038167Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:28:47.451445Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:28:49.020020Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:29:06.347459Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:29:07.987050Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:29:10.895484Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:29:19.099441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:29:23.584698Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-02T22:29:23.578Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 71 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "240022b8-b987-4edc-8af9-dcb8b7efe4fe" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:50676 10.133.0.36:443 10.133.0.11:48338 - v1-completions-path [2026-11-02T22:29:23.823Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 56 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "822bc716-0150-4303-af7a-3457286b1da9" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:50676 10.133.0.36:443 10.133.0.11:48344 - v1-completions-path 2026-11-02T22:29:24.325473Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-02T22:29:24.271Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 57 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "60d3ffce-b07a-47fb-8242-7f771d22422d" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:50682 10.133.0.36:443 10.133.0.11:48356 - v1-completions-path 2026-11-02T22:29:40.304829Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:29:50.403858Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:29:52.015152Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-02T22:29:59.196Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 59 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "a011ed63-c940-4ff2-9968-8bce09389ee7" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:50682 10.133.0.36:443 10.133.0.11:54200 - v1-completions-path [2026-11-02T22:29:59.427Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 56 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "c5e33a78-6898-47a1-b858-8dda6b4ae969" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:50682 10.133.0.36:443 10.133.0.11:54214 - v1-completions-path 2026-11-02T22:30:10.996116Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:30:12.632358Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:30:29.204932Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:30:31.297781Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:30:47.917943Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:30:49.610186Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:31:05.564708Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:31:16.302461Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:31:19.107281Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:31:22.746758Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:31:22.746793Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:31:22.746811Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:31:22.746817Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T22:31:22.682Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 324 64 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "ef682d48-0c37-4e04-8a63-558202995e88" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.36:58586 10.133.0.36:443 10.133.0.11:52580 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T22:31:22.802Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 295 55 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "1ea39e9c-33d3-442c-aaf6-44b452d2dd1a" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:51282 10.133.0.36:443 10.133.0.11:52584 - v1-completions-path [2026-11-02T22:31:37.408Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 32 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "0634415a-5939-4408-a467-f6c0a21a70a3" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.36:443 10.133.0.11:38348 - v1-completions-path [2026-11-02T22:31:38.954Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 0 31 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "0cde7c7b-c2e6-4e1f-8f53-ac79dd392473" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.36:443 10.133.0.11:38358 - v1-completions-path 2026-11-02T22:31:41.313949Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:31:56.552945Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:32:06.752151Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:32:22.161964Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:32:24.416089Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:34:01.008731Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:34:03.353134Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:35:27.858290Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:35:42.574758Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:35:44.299471Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:36:01.553341Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:37:27.371501Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:37:45.095219Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:38:00.144851Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:38:11.607703Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:38:14.973914Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:38:26.111615Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:38:28.461131Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-02T22:38:34.550Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 67 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "9b81c2ac-2c58-404e-914d-0fc7a1766553" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:50682 10.133.0.36:443 10.133.0.11:58070 - v1-completions-path 2026-11-02T22:38:46.353858Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:41:34.231992Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:41:37.104009Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-02T22:41:37.014Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 89 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "55edf3d6-7a33-4829-8e41-977a39ccf7d2" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.134.0.30:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:41740 10.133.0.36:443 10.133.0.11:34112 - v1-catch-all-path 2026-11-02T22:41:39.341600Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:41:42.727750Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-02T22:41:42.667Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 60 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "02056ed0-078e-456c-9cf4-f6760155379a" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.134.0.30:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:41740 10.133.0.36:443 10.133.0.11:39688 - v1-catch-all-path 2026-11-02T22:41:54.352923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:41:55.911971Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:41:58.847229Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-02T22:41:58.783Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 63 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "f46ccecb-7f76-4b42-9ef2-064b9a1a8957" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.134.0.30:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.36:41740 10.133.0.36:443 10.133.0.11:54170 - v1-catch-all-path 2026-11-02T22:42:10.363613Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-02T22:42:21.048078Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7