2026-11-03T20:50:17.856685Z info FLAG: --concurrency="0" 2026-11-03T20:50:17.856701Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T20:50:17.856705Z info FLAG: --help="false" 2026-11-03T20:50:17.856707Z info FLAG: --log_as_json="false" 2026-11-03T20:50:17.856709Z info FLAG: --log_caller="" 2026-11-03T20:50:17.856711Z info FLAG: --log_output_level="default:info" 2026-11-03T20:50:17.856713Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T20:50:17.856717Z info FLAG: --log_target="[stdout]" 2026-11-03T20:50:17.856719Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T20:50:17.856721Z info FLAG: --outlierLogPath="" 2026-11-03T20:50:17.856723Z info FLAG: --profiling="true" 2026-11-03T20:50:17.856725Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T20:50:17.856726Z info FLAG: --proxyLogLevel="warning" 2026-11-03T20:50:17.856729Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T20:50:17.856731Z info FLAG: --stsPort="0" 2026-11-03T20:50:17.856732Z info FLAG: --templateFile="" 2026-11-03T20:50:17.856734Z info FLAG: --tokenManagerPlugin="" 2026-11-03T20:50:17.856736Z info FLAG: --vklog="0" 2026-11-03T20:50:17.856739Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T20:50:17.856870Z info Proxy role ips=[10.133.0.30] type=router id=maas-default-gateway-openshift-default-7bdc57c4cf-cws5p.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T20:50:17.856912Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T20:50:17.857939Z info cpu limit detected as 2, setting concurrency 2026-11-03T20:50:17.858116Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T20:50:17.858127Z info JWT policy is third-party-jwt 2026-11-03T20:50:17.858130Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T20:50:17.860809Z info Starting default Istio SDS Server 2026-11-03T20:50:17.860830Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T20:50:17.860848Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T20:50:17.860932Z info Opening status port 15020 2026-11-03T20:50:17.861459Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T20:50:17.861700Z info sds Starting SDS grpc server 2026-11-03T20:50:17.861723Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T20:50:17.862788Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T20:50:17.863499Z info Starting proxy agent 2026-11-03T20:50:17.863528Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T20:50:17.924395Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T20:50:18.006189Z info cache generated new workload certificate resourceName=default latency=144.368264ms ttl=23h59m59.993815092s 2026-11-03T20:50:18.006247Z info cache Root cert has changed, start rotating root cert 2026-11-03T20:50:18.006284Z info cache returned workload trust anchor from cache ttl=23h59m59.993717071s 2026-11-03T20:50:18.068616Z info ads ADS: new connection for node:1 2026-11-03T20:50:18.068704Z info cache returned workload certificate from cache ttl=23h59m59.931297628s 2026-11-03T20:50:18.069616Z info ads ADS: new connection for node:2 2026-11-03T20:50:18.069711Z info cache returned workload trust anchor from cache ttl=23h59m59.930290678s 2026-11-03T20:50:18.148092Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T20:50:24.292568Z error failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused 2026-11-03T20:50:25.951905Z info Readiness succeeded in 8.098065697s 2026-11-03T20:50:25.952263Z info Envoy proxy is ready 2026-11-03T20:50:31.536575Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:50:31.536615Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:50:31.536646Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:50:31.536652Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T20:50:31.363Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 55 320 172 - "44.212.242.249,10.132.0.9" "curl/7.76.1" "329db75a-ec31-4c03-9452-80e66be640b1" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.30:44298 10.133.0.30:443 10.132.0.9:40034 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:55:38.079125Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:55:38.079171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:55:38.079189Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:55:38.079196Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T20:55:37.952Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 84 327 126 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "aeb576d6-2ca5-4daf-9e9f-994f4a2b4caf" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.30:54016 10.133.0.30:443 10.132.0.9:37334 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T20:55:39.773Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 88 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "5305bcc8-1c59-4279-bc96-37bf1a95e234" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.30:443 10.132.0.9:37346 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:55:40.176198Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:55:40.176234Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:55:40.176250Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T20:55:40.176256Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T20:55:40.064Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 111 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "1385af9c-a341-4705-b1f2-f2c0d2030d21" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.30:35076 10.133.0.30:443 10.132.0.9:37360 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:55:41.175617Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:55:41.175680Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:55:41.175697Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:55:41.175703Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T20:55:40.928Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 247 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "6062a96d-f16f-4b63-8c03-37dda62a410e" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.30:54016 10.133.0.30:443 10.132.0.9:37390 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:55:43.011559Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:55:43.011591Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:55:43.011602Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T20:55:43.011607Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T20:55:41.913Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 248 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "e9d3caca-d209-44c3-bb82-ae11cefd31ba" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.30:443 10.132.0.9:33388 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T20:55:42.883Z] "DELETE /v1/api-keys/4beed25a-af71-4eec-b066-12f183a12a9b HTTP/1.1" 200 - via_upstream - "-" 0 321 128 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "e03f1668-a1c5-4c68-b3fe-99bf446529a3" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.30:35076 10.133.0.30:443 10.132.0.9:33390 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T20:55:45.760Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 85 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "a6107faa-9c1e-40af-b786-a1cdcbe971cc" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.30:443 10.132.0.9:33402 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:55:56.038923Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:55:56.038960Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:55:56.038974Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T20:55:56.038980Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T20:55:56.275458Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T20:55:56.275482Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T20:55:56.275485Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T20:55:56.275489Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-03T20:55:55.926Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 112 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "d37d34de-0ddb-495d-a174-fef346b99609" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.30:54016 10.133.0.30:443 10.132.0.9:52704 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T20:55:56.236Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 77 38 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "e9aef6c2-3f9c-40e8-9dc5-538169f98f41" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.30:443 10.132.0.9:52718 - v1-completions-path 2026-11-03T20:56:05.112211Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T20:56:20.299120Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T21:02:06.614Z] "POST /maas-api/v1/tokens HTTP/1.1" 401 - - "-" 20 88 24 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "b6a34d09-c433-4ff2-80d6-c5d8a3bd864f" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.30:443 10.132.0.9:38622 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T21:02:39.271576Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:02:39.271610Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:02:39.271623Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T21:02:39.271645Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T21:02:39.253Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 18 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "4d5fa6f9-1cc3-4c09-bf5e-2e2e23f8ed83" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.30:36616 10.133.0.30:443 10.132.0.9:45498 - odh-ai-gateway-infra.maas-api-route.4 2026-11-03T21:03:32.000857Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:06:26.466043Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:06:58.337237Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:08:40.540258Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:15:06.946832Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:15:19.056164Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:16:09.916929Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:16:31.403483Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:16:36.273559Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:17:02.724191Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:17:19.695786Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T21:17:30.027Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 73 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "8957634e-da71-40b8-94fc-3334d224e291" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.30:55864 10.133.0.30:443 10.132.0.9:36708 - v1-completions-path 2026-11-03T21:17:50.617399Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:18:01.158894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:18:07.686800Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T21:18:21.149Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 62 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "1885aed6-6f9d-4d7b-923a-0f134e6b4591" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.30:55864 10.133.0.30:443 10.132.0.9:57940 - v1-completions-path 2026-11-03T21:18:22.269555Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-03T21:18:22.206Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "ddafad78-6a1c-402d-bf87-ecb28eaf7275" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.30:55864 10.133.0.30:443 10.132.0.9:59340 - v1-completions-path 2026-11-03T21:18:36.692448Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:18:41.422964Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:19:04.788441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:19:11.268601Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:19:35.444865Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:19:47.253842Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:20:07.690064Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:20:20.854887Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:20:26.713528Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:20:52.760798Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:21:03.951510Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:21:09.759567Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:21:09.759600Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:21:09.759612Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:21:09.759617Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T21:21:09.646Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 89 331 112 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "4859641e-459f-42f3-89fa-f50791c6bc7e" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.30:59100 10.133.0.30:443 10.132.0.9:33620 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T21:21:09.962Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 287 75 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "ae35a17e-1876-4e2a-9a84-29e5dc5b18c6" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.30:53030 10.133.0.30:443 10.132.0.9:33636 - v1-completions-path 2026-11-03T21:21:14.873668Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-03T21:21:20.677790Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T21:21:29.382Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 298 67 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "058c1acb-4d52-4208-975f-a9690c808182" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.30:59662 10.133.0.30:443 10.132.0.9:44388 - v1-completions-path 2026-11-03T21:21:43.528060Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:21:54.152560Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:22:14.231349Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:22:20.255068Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:24:54.597743Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:25:00.714143Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:26:17.151476Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T21:26:17.151500Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T21:26:17.151503Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T21:26:17.151507Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T21:26:17.107Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 403 - - "-" 73 113 44 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "715f2982-662f-4d5c-b7d4-6de0ce5cbea7" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.30:443 10.132.0.9:33804 - v1-completions-path 2026-11-03T21:26:39.352830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:26:56.790831Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:27:01.777913Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:27:12.103790Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T21:27:12.103813Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T21:27:12.103816Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T21:27:12.103820Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T21:27:12.067Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 35 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "a20b1870-d255-4a3c-9365-740c321f679e" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.30:443 10.132.0.9:43408 - v1-completions-path 2026-11-03T21:27:25.789822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:29:33.739965Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:30:18.849811Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:30:46.173786Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:31:00.192280Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:31:17.904124Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:31:30.353798Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:31:44.769058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T21:31:53.338Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 62 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "7a5c3af0-3935-4117-8819-2eaace39b75c" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.30:55864 10.133.0.30:443 10.132.0.9:49640 - v1-completions-path [2026-11-03T21:31:53.872Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 56 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "da246861-8d48-45b4-93b7-11e05c9d60e9" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.30:443 10.132.0.9:49650 - v1-completions-path 2026-11-03T21:32:07.963094Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:35:38.705327Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:35:47.816434Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T21:35:47.741Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 75 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "1bb47f77-3ba8-461d-8233-804f839970f3" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.134.0.36:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.30:42454 10.133.0.30:443 10.132.0.9:56168 - v1-catch-all-path 2026-11-03T21:35:52.353429Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:36:13.278385Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:36:18.214253Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:36:18.732212Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T21:36:18.732802Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T21:36:18.627Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 105 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "6a363a38-44d7-42b0-aed5-96776b79fbce" "maas.apps.a89677555ca38b64ed7f.prod.konflux-ocp-ci.dev" "10.134.0.36:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.30:42454 10.133.0.30:443 10.132.0.9:57110 - v1-catch-all-path 2026-11-03T21:36:32.659824Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T21:36:44.177899Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9