2026-11-02T02:56:09.841199Z info FLAG: --concurrency="0" 2026-11-02T02:56:09.841255Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T02:56:09.841263Z info FLAG: --help="false" 2026-11-02T02:56:09.841267Z info FLAG: --log_as_json="false" 2026-11-02T02:56:09.841271Z info FLAG: --log_caller="" 2026-11-02T02:56:09.841283Z info FLAG: --log_output_level="default:info" 2026-11-02T02:56:09.841287Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T02:56:09.841295Z info FLAG: --log_target="[stdout]" 2026-11-02T02:56:09.841300Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T02:56:09.841303Z info FLAG: --outlierLogPath="" 2026-11-02T02:56:09.841314Z info FLAG: --profiling="true" 2026-11-02T02:56:09.841318Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T02:56:09.841322Z info FLAG: --proxyLogLevel="warning" 2026-11-02T02:56:09.841325Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T02:56:09.841336Z info FLAG: --stsPort="0" 2026-11-02T02:56:09.841340Z info FLAG: --templateFile="" 2026-11-02T02:56:09.841343Z info FLAG: --tokenManagerPlugin="" 2026-11-02T02:56:09.841349Z info FLAG: --vklog="0" 2026-11-02T02:56:09.841354Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T02:56:09.886966Z info Proxy role ips=[10.132.1.143] type=router id=maas-default-gateway-openshift-default-769df7846c-x26br.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T02:56:09.887117Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T02:56:09.888420Z info cpu limit detected as 2, setting concurrency 2026-11-02T02:56:09.888674Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T02:56:09.888691Z info JWT policy is third-party-jwt 2026-11-02T02:56:09.888718Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T02:56:09.890968Z info Starting default Istio SDS Server 2026-11-02T02:56:09.891057Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T02:56:09.891113Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T02:56:09.890953Z info Opening status port 15020 2026-11-02T02:56:09.891861Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T02:56:09.895103Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T02:56:09.899179Z info Starting proxy agent 2026-11-02T02:56:09.899273Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T02:56:09.898537Z info sds Starting SDS grpc server 2026-11-02T02:56:09.915453Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T02:56:09.982966Z info cache generated new workload certificate resourceName=default latency=91.366445ms ttl=23h59m59.017037997s 2026-11-02T02:56:09.983084Z info cache Root cert has changed, start rotating root cert 2026-11-02T02:56:09.983159Z info cache returned workload trust anchor from cache ttl=23h59m59.016841072s 2026-11-02T02:56:10.003575Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T02:56:10.178876Z info ads ADS: new connection for node:1 2026-11-02T02:56:10.178967Z info cache returned workload certificate from cache ttl=23h59m58.821035721s 2026-11-02T02:56:10.180130Z info ads ADS: new connection for node:2 2026-11-02T02:56:10.180304Z info cache returned workload trust anchor from cache ttl=23h59m58.819697834s 2026-11-02T02:56:10.180972Z info ads ADS: new connection for node:3 2026-11-02T02:56:10.181419Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T02:56:10.181476Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T02:56:10.297189Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T02:56:18.818841Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T02:56:19.747852Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-02T02:56:19.985776Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-02T02:56:21.256283Z info Readiness succeeded in 11.424814844s 2026-11-02T02:56:21.256796Z info Envoy proxy is ready 2026-11-02T02:56:58.459452Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T02:57:22.208498Z info ads ADS: "" 3 terminated 2026-11-02T02:57:22.521686Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T02:57:27.185817Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T02:57:43.546566Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:00:26.527156Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:00:26.527212Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:00:26.527235Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:00:26.527242Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T03:00:26.397Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 129 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "8a8ec667-ce5f-45f9-8f1f-4400f90ece85" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.143:39712 10.132.1.143:443 10.132.0.10:59354 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T03:00:28.685272Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:00:28.685308Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:00:28.685320Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:00:28.685325Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T03:00:28.632Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 52 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "b68cecae-c5e3-4038-a2ef-e03cced0ce25" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.143:39712 10.132.1.143:443 10.132.0.10:49618 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T03:00:36.632766Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:01:04.768008Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:01:30.265868Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:01:45.868715Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:01:59.571363Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:02:12.176767Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:02:17.124061Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T03:02:21.377Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 77 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "c537aec6-c502-4d3c-803a-5bfdde9d1084" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:48796 10.132.1.143:443 10.132.0.10:48454 - v1-completions-path 2026-11-02T03:02:36.327975Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:03:23.645084Z info ads ADS: new connection for node:4 2026-11-02T03:03:23.650354Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T03:03:23.650410Z info cache read certificate from file resource=file-root:system 2026-11-02T03:03:26.857289Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:03:29.914860Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:03:29.922494Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T03:03:29.922555Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T03:03:29.922581Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T03:03:29.922589Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T03:03:29.743Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 179 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "023334b5-9887-44b6-a81c-f4f044382309" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.143:38436 10.132.1.143:443 10.132.0.10:45370 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T03:03:51.027967Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T03:03:51.027999Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T03:03:51.028009Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T03:03:51.028013Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T03:03:51.028159Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T03:03:50.834Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1717 194 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "b0199068-010d-4e79-9099-cf3571584302" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.143:53692 10.132.1.143:443 10.132.0.10:44580 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T03:04:09.962040Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:04:09.962073Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:04:09.962084Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:04:09.962089Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T03:04:09.962258Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T03:04:09.766Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1706 195 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "a66e20d2-67b5-4f3d-a0c8-011dde1de076" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.143:48972 10.132.1.143:443 10.132.0.10:46956 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T03:04:12.279737Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:04:12.279769Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:04:12.279778Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T03:04:12.279783Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:04:12.279943Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T03:04:12.154Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1715 125 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "9545b6fa-2f8d-4a4a-9db2-0f6b477f0cac" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.143:48972 10.132.1.143:443 10.132.0.10:46974 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T03:04:19.204998Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-02T03:04:19.301223Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:04:19.301255Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:04:19.301264Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:04:19.301269Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T03:04:19.301439Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T03:04:19.150Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 151 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "31a1b05e-d59f-96f3-a461-4047a1360f34" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.143:48972 10.132.1.143:443 10.132.0.10:48698 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T03:05:07.305394Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:05:07.325328Z info ads ADS: "" 4 terminated 2026-11-02T03:05:09.972390Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:05:20.516240Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:05:38.552954Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:05:43.168782Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T03:05:46.381Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 297 67 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "77f20a0b-755f-4cc8-8a3c-27d599f666d9" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:34212 10.132.1.143:443 10.132.0.10:59946 - v1-completions-path [2026-11-02T03:06:01.841Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 75 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "1d059a40-19af-40a6-97fe-cb841e487d33" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:54952 10.132.1.143:443 10.132.0.10:57716 - v1-completions-path 2026-11-02T03:06:03.300427Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:06:21.786288Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:06:32.978671Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-02T03:06:32.917Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 64 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "5e63f051-dcc8-4d68-b094-948e84111065" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:48796 10.132.1.143:443 10.132.0.10:45636 - v1-completions-path [2026-11-02T03:06:34.487Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 56 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "c5477060-b981-4815-be53-2136288f3d65" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:34212 10.132.1.143:443 10.132.0.10:45644 - v1-completions-path [2026-11-02T03:06:36.057Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "7dbff29f-72c8-4f43-92e1-ebebc614a976" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:34212 10.132.1.143:443 10.132.0.10:45662 - v1-completions-path [2026-11-02T03:06:36.856Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 305 60 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "60f026a8-07a4-4380-ace2-a8a0b553c52c" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:34212 10.132.1.143:443 10.132.0.10:45668 - v1-completions-path 2026-11-02T03:06:53.666962Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:07:04.641769Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:07:10.754759Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T03:07:23.361Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 59 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "7e5c77d0-95ab-4b2e-aa61-17bcff8aa2f6" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:34212 10.132.1.143:443 10.132.0.10:56332 - v1-completions-path [2026-11-02T03:07:25.123Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 54 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "d1e148f5-5873-4fe7-ac3a-47c652c0caf2" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:34212 10.132.1.143:443 10.132.0.10:56340 - v1-completions-path 2026-11-02T03:07:38.904267Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:07:43.443632Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:07:56.729721Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:08:05.501901Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:08:13.385689Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:08:23.310230Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:08:32.779801Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:08:51.997930Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:09:12.612264Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:09:12.612304Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T03:09:12.612322Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T03:09:12.612327Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T03:09:12.494Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 89 331 118 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "8b928362-0a81-44cf-9765-38c2b966babe" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.143:51400 10.132.1.143:443 10.132.0.10:38796 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T03:09:22.994603Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:09:45.732133Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:09:56.510847Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T03:12:43.178Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 51 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "2266441a-f82c-4ae3-838d-74272e1a5cbb" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.143:443 10.132.0.10:40576 - v1-completions-path 2026-11-02T03:14:15.758783Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:14:18.623731Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: gRPC status code is not OK thread=24 [2026-11-02T03:14:16.590Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 500 - - "-" 73 23 2034 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "51990e7f-6b8b-4be7-a17f-6f6450ef99db" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.143:443 10.132.0.10:46734 - v1-completions-path 2026-11-02T03:14:35.533669Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:14:40.201384Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:14:40.902590Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T03:14:40.902644Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T03:14:40.902668Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T03:14:40.902676Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T03:14:40.765Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 65 308 137 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "dc5676dd-5f21-483e-8e9a-e33ccde0bd04" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.143:34274 10.132.1.143:443 10.132.0.10:37408 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T03:14:53.175523Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:15:09.489787Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:15:14.797777Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T03:15:18.802Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 61 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "36e652e6-73d3-4848-9ccb-be7044e992df" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:48796 10.132.1.143:443 10.132.0.10:36576 - v1-completions-path [2026-11-02T03:15:19.874Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 60 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "0327e410-66f9-4812-9ce0-da360f9398d7" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:34212 10.132.1.143:443 10.132.0.10:36588 - v1-completions-path 2026-11-02T03:15:34.902376Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:15:45.536279Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:15:51.836884Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:16:21.862095Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:16:28.077215Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:16:34.537321Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T03:16:43.900Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 68 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "3120ce51-3a1b-4f33-9268-dc0b9801b701" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:48796 10.132.1.143:443 10.132.0.10:52976 - v1-completions-path [2026-11-02T03:16:51.443Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 66 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "adb3cad0-43ae-4447-b890-136e58b59d95" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:48796 10.132.1.143:443 10.132.0.10:36470 - v1-completions-path 2026-11-02T03:17:06.266879Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:17:17.243268Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:17:23.800963Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:17:30.084545Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:17:41.697094Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T03:17:48.132Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 63 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "25d83e47-31e8-4d48-a401-46b37f10ee66" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:48796 10.132.1.143:443 10.132.0.10:52730 - v1-completions-path 2026-11-02T03:18:04.936329Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:18:15.634120Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:18:20.596961Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:18:38.287438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:18:44.718924Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:19:08.033079Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:19:30.340915Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:19:52.037238Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:20:12.153016Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:20:23.154777Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:20:33.863376Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:20:44.693738Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:20:55.488419Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T03:21:06.045071Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T03:21:14.409Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 65 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "a4cc29d2-a02f-44b9-b1f3-dd2d6801fed7" "maas.apps.82bac16f36aa9aea1c00.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.143:54952 10.132.1.143:443 10.132.0.10:47634 - v1-completions-path