2026-11-03T03:17:48.508663Z info FLAG: --concurrency="0" 2026-11-03T03:17:48.508679Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T03:17:48.508683Z info FLAG: --help="false" 2026-11-03T03:17:48.508686Z info FLAG: --log_as_json="false" 2026-11-03T03:17:48.508688Z info FLAG: --log_caller="" 2026-11-03T03:17:48.508690Z info FLAG: --log_output_level="default:info" 2026-11-03T03:17:48.508692Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T03:17:48.508696Z info FLAG: --log_target="[stdout]" 2026-11-03T03:17:48.508699Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T03:17:48.508701Z info FLAG: --outlierLogPath="" 2026-11-03T03:17:48.508703Z info FLAG: --profiling="true" 2026-11-03T03:17:48.508705Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T03:17:48.508707Z info FLAG: --proxyLogLevel="warning" 2026-11-03T03:17:48.508709Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T03:17:48.508712Z info FLAG: --stsPort="0" 2026-11-03T03:17:48.508714Z info FLAG: --templateFile="" 2026-11-03T03:17:48.508716Z info FLAG: --tokenManagerPlugin="" 2026-11-03T03:17:48.508723Z info FLAG: --vklog="0" 2026-11-03T03:17:48.508726Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T03:17:48.508859Z info Proxy role ips=[10.134.1.39] type=router id=maas-default-gateway-openshift-default-5bc884dd87-bpbq7.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T03:17:48.508901Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T03:17:48.509922Z info cpu limit detected as 2, setting concurrency 2026-11-03T03:17:48.510551Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T03:17:48.510562Z info JWT policy is third-party-jwt 2026-11-03T03:17:48.510565Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T03:17:48.512091Z info Starting default Istio SDS Server 2026-11-03T03:17:48.512115Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T03:17:48.512147Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T03:17:48.512657Z info Opening status port 15020 2026-11-03T03:17:48.514035Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T03:17:48.515885Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T03:17:48.517119Z info Starting proxy agent 2026-11-03T03:17:48.517164Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T03:17:48.524793Z info sds Starting SDS grpc server 2026-11-03T03:17:48.524811Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T03:17:48.593705Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T03:17:48.622927Z info cache generated new workload certificate resourceName=default latency=108.924985ms ttl=23h59m59.377076488s 2026-11-03T03:17:48.622959Z info cache Root cert has changed, start rotating root cert 2026-11-03T03:17:48.622999Z info cache returned workload trust anchor from cache ttl=23h59m59.377001377s 2026-11-03T03:17:48.739768Z info ads ADS: new connection for node:1 2026-11-03T03:17:48.739836Z info cache returned workload certificate from cache ttl=23h59m59.260166119s 2026-11-03T03:17:48.740301Z info ads ADS: new connection for node:2 2026-11-03T03:17:48.740421Z info cache returned workload trust anchor from cache ttl=23h59m59.259581227s 2026-11-03T03:17:48.861612Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:17:56.788311Z info Readiness succeeded in 8.281982988s 2026-11-03T03:17:56.788737Z info Envoy proxy is ready 2026-11-03T03:17:57.558385Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:18:10.368085Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:18:20.071523Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:18:36.638605Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:18:48.421577Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:18:53.013171Z info ads ADS: new connection for node:3 2026-11-03T03:18:53.013339Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T03:18:53.013383Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T03:18:57.370389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:19:40.021447Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:20:05.202519Z info ads ADS: "" 3 terminated 2026-11-03T03:20:05.258404Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:20:10.284Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 81 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "213ff9f0-1a2c-4d16-84ce-aa0a8dfcf43b" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:33872 10.134.1.39:443 10.132.0.20:55766 - v1-completions-path 2026-11-03T03:20:11.367236Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:20:27.076371Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:23:23.054737Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:23:47.505741Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:24:13.560661Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:24:28.861560Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:24:37.077440Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:24:49.940768Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:24:54.559082Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:25:12.644837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:26:01.081090Z info ads ADS: new connection for node:4 2026-11-03T03:26:01.086313Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T03:26:01.086381Z info cache read certificate from file resource=file-root:system 2026-11-03T03:26:05.046271Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:26:06.045173Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:26:28.025592Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:26:28.025623Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:26:28.025633Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T03:26:28.025637Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:26:28.025777Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T03:26:27.814Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1719 211 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "563eb80a-3fc9-4a21-8e2e-f5fd6390f420" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.1.39:34774 10.134.1.39:443 10.132.0.20:44006 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T03:26:32.755757Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:26:32.755787Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:26:32.755797Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T03:26:32.755802Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:26:32.755939Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T03:26:32.629Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1717 126 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "4a556814-03c2-4da9-a008-92b4ad682058" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.1.39:34774 10.134.1.39:443 10.132.0.20:51088 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T03:26:59.569416Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T03:26:59.569443Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T03:26:59.569452Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-03T03:26:59.569457Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T03:26:59.569614Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-11-03T03:26:59.375Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1720 194 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "9af55516-ab59-4c75-a9be-aeec8f09f504" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.1.39:39074 10.134.1.39:443 10.132.0.20:38598 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T03:27:18.303106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T03:27:18.303679Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:27:18.303704Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:27:18.303714Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T03:27:18.303718Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:27:18.303890Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T03:27:18.175Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1715 128 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e3d786fb-f302-4308-b339-5918964e76dd" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.1.39:34774 10.134.1.39:443 10.132.0.20:34058 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T03:27:25.410219Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:27:25.410252Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:27:25.410262Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T03:27:25.410266Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:27:25.410448Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T03:27:25.281Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1718 129 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "5e9d78f9-50d4-4afe-8d6e-de22a6020ec1" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.1.39:34774 10.134.1.39:443 10.132.0.20:33182 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T03:28:00.308325Z info ads ADS: "" 4 terminated 2026-11-03T03:28:00.353943Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:28:02.794714Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:28:04.844239Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-03T03:28:04.844262Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-03T03:28:04.844265Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-03T03:28:04.844269Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=26 [2026-11-03T03:28:04.807Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 37 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e872dc8b-7a19-49e4-a497-d39370f0ba28" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.39:443 10.132.0.20:36662 - v1-completions-path 2026-11-03T03:28:13.125216Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:28:17.649Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 74 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "f739f5eb-328c-4ada-a2cf-33ef4ae1dce8" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:33350 10.134.1.39:443 10.132.0.20:39926 - v1-completions-path 2026-11-03T03:28:31.629493Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:28:36.009763Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:28:38.927Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 63 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "07ebce16-b88f-48e6-841e-057025c0a045" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:49656 10.134.1.39:443 10.132.0.20:48936 - v1-completions-path [2026-11-03T03:28:40.242Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 299 57 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "5b3b0d5a-b016-4d97-b111-fe3b33c6593b" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:33350 10.134.1.39:443 10.132.0.20:34122 - v1-completions-path [2026-11-03T03:28:42.042Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 52 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "1552b1bd-9d4c-4181-ad83-e73a2a6bd988" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.39:443 10.132.0.20:34140 - v1-completions-path 2026-11-03T03:28:55.745844Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:29:12.703652Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:29:21.657052Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 [2026-11-03T03:29:21.651Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 61 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "6fbdbc14-ca67-460a-b424-bd6a62dad8c9" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.39:443 10.132.0.20:55044 - v1-completions-path [2026-11-03T03:29:23.516Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 57 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "586beace-1826-4ba1-80df-52b7bf8747f0" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:33350 10.134.1.39:443 10.132.0.20:55054 - v1-completions-path [2026-11-03T03:29:25.305Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 57 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "a2925c1b-491e-477a-8080-d29a5b74ad87" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:49656 10.134.1.39:443 10.132.0.20:55068 - v1-completions-path 2026-11-03T03:29:43.647047Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:29:54.291143Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:30:00.356221Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:30:12.837Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 297 58 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "c1d91fee-67f4-4c5a-a6b5-5c12033215d7" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:33350 10.134.1.39:443 10.132.0.20:33438 - v1-completions-path 2026-11-03T03:30:30.320343Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:30:34.953188Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:30:47.559928Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:30:50.538Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 281 70 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "de4eb3be-7bd2-482f-b46a-2e9b0fbaba2e" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:45120 10.134.1.39:443 10.132.0.20:58436 - v1-completions-path 2026-11-03T03:30:56.206736Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:31:05.174623Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:31:13.995467Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:31:22.938180Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:31:41.063121Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:31:55.313Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 37 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "03bd8d6e-dab9-4d68-af6d-4724b615bdbe" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.39:443 10.132.0.20:50978 - v1-completions-path 2026-11-03T03:32:02.608308Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:32:02.608343Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:32:02.608384Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T03:32:02.608390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T03:32:02.500Z] "DELETE /v1/api-keys/097509a8-858a-4a88-8537-695fc701705a HTTP/1.1" 404 - via_upstream - "-" 0 29 107 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e0679a8a-b500-46bd-8a64-0692ace1b762" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.39:44228 10.134.1.39:443 10.132.0.20:60190 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T03:32:12.402346Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:32:32.715564Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:32:43.421799Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:35:35.775422Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:37:07.345894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:37:11.093563Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T03:37:11.093604Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T03:37:11.093622Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-03T03:37:11.093628Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-11-03T03:37:10.972Z] "DELETE /v1/api-keys/e2e-trlp-test-key HTTP/1.1" 404 - via_upstream - "-" 0 29 120 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "223a1cfe-44c9-411f-9141-a892a60fd106" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.39:34796 10.134.1.39:443 10.132.0.20:34832 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T03:37:24.530139Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:37:29.338998Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:37:32.171Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 117 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e5613eec-bb21-4e4e-aeac-ca7f2303a807" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:45120 10.134.1.39:443 10.132.0.20:33118 - v1-completions-path 2026-11-03T03:37:41.919331Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:37:59.468613Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:38:05.896056Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:38:10.336Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 63 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "96b5963b-f208-409a-8b4a-aa17418b1680" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:49656 10.134.1.39:443 10.132.0.20:52412 - v1-completions-path [2026-11-03T03:38:10.688Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 65 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e3abaf5c-ee07-4dec-a68c-f36d3b63a017" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:33350 10.134.1.39:443 10.132.0.20:52414 - v1-completions-path 2026-11-03T03:38:25.872701Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:38:36.602328Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:38:42.749894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:38:54.059Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 65 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "bbe57e48-fa6e-4d39-9256-8d5a58a4260c" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:49656 10.134.1.39:443 10.132.0.20:33234 - v1-completions-path [2026-11-03T03:38:54.338Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 59 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "a899ca7a-dd1d-4a00-a817-88b4d951dc60" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:49656 10.134.1.39:443 10.132.0.20:33246 - v1-completions-path 2026-11-03T03:39:10.087465Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:39:16.275490Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:39:21.922894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:39:34.577Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "a0698ea8-b467-485e-87ed-6963d300e5e2" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:49656 10.134.1.39:443 10.132.0.20:53532 - v1-completions-path [2026-11-03T03:39:35.998Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 60 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "cd16b74c-5a2a-4d66-8a82-5d529b7f32a0" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:33350 10.134.1.39:443 10.132.0.20:53558 - v1-completions-path 2026-11-03T03:39:52.956300Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:40:03.817376Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:40:10.180213Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:40:15.981677Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:40:26.278691Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T03:40:33.459Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 63 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e0aa3083-41de-40ce-bd65-45447cad118d" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.1.39:33350 10.134.1.39:443 10.132.0.20:57994 - v1-completions-path [2026-11-03T03:40:34.913Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 56 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "7e7a954a-271d-46d3-b659-2e1128aa9f27" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.1.39:443 10.132.0.20:57998 - v1-completions-path 2026-11-03T03:40:35.273156Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T03:40:35.273205Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T03:40:35.273227Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T03:40:35.273234Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-03T03:40:35.160Z] "DELETE /v1/api-keys/8b07f5cd-41bd-453b-a1c0-b9d3b38501e2 HTTP/1.1" 200 - via_upstream - "-" 0 367 112 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e564f12b-6092-42f2-8816-0caa7a422ddf" "maas.apps.abf4b066b9836acb0417.prod.konflux-ocp-ci.dev" "10.132.0.151:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.1.39:58330 10.134.1.39:443 10.132.0.20:58002 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T03:40:48.817960Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:40:59.598120Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:41:04.382393Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:41:20.311690Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:41:26.265747Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:41:50.132987Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:42:12.182349Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:42:32.455973Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:42:45.118954Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:43:05.013436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:43:15.686847Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:43:26.545969Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:43:37.335157Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T03:43:47.868164Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9