2026-10-31T04:32:08.825213Z info FLAG: --concurrency="0" 2026-10-31T04:32:08.825234Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-31T04:32:08.825240Z info FLAG: --help="false" 2026-10-31T04:32:08.825244Z info FLAG: --log_as_json="false" 2026-10-31T04:32:08.825246Z info FLAG: --log_caller="" 2026-10-31T04:32:08.825249Z info FLAG: --log_output_level="default:info" 2026-10-31T04:32:08.825252Z info FLAG: --log_stacktrace_level="default:none" 2026-10-31T04:32:08.825258Z info FLAG: --log_target="[stdout]" 2026-10-31T04:32:08.825262Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-31T04:32:08.825264Z info FLAG: --outlierLogPath="" 2026-10-31T04:32:08.825268Z info FLAG: --profiling="true" 2026-10-31T04:32:08.825271Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-31T04:32:08.825274Z info FLAG: --proxyLogLevel="warning" 2026-10-31T04:32:08.825277Z info FLAG: --serviceCluster="istio-proxy" 2026-10-31T04:32:08.825280Z info FLAG: --stsPort="0" 2026-10-31T04:32:08.825283Z info FLAG: --templateFile="" 2026-10-31T04:32:08.825285Z info FLAG: --tokenManagerPlugin="" 2026-10-31T04:32:08.825289Z info FLAG: --vklog="0" 2026-10-31T04:32:08.825293Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-31T04:32:08.825468Z info Proxy role ips=[10.132.1.122] type=router id=maas-default-gateway-openshift-default-6886b96f9-lqmhm.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-31T04:32:08.825570Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-31T04:32:08.827133Z info cpu limit detected as 2, setting concurrency 2026-10-31T04:32:08.827351Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-31T04:32:08.827365Z info JWT policy is third-party-jwt 2026-10-31T04:32:08.827369Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-31T04:32:08.829066Z info Starting default Istio SDS Server 2026-10-31T04:32:08.829092Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-31T04:32:08.829109Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-31T04:32:08.829308Z info Opening status port 15020 2026-10-31T04:32:08.830183Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-31T04:32:08.830827Z info sds Starting SDS grpc server 2026-10-31T04:32:08.830845Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-31T04:32:08.831420Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-31T04:32:08.832170Z info Starting proxy agent 2026-10-31T04:32:08.832205Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-31T04:32:08.896057Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-31T04:32:08.956202Z info cache generated new workload certificate resourceName=default latency=125.273763ms ttl=23h59m59.043802797s 2026-10-31T04:32:08.956235Z info cache Root cert has changed, start rotating root cert 2026-10-31T04:32:08.956272Z info cache returned workload trust anchor from cache ttl=23h59m59.043728795s 2026-10-31T04:32:09.057369Z info ads ADS: new connection for node:1 2026-10-31T04:32:09.058064Z info cache returned workload certificate from cache ttl=23h59m58.941944648s 2026-10-31T04:32:09.058107Z info ads ADS: new connection for node:2 2026-10-31T04:32:09.058416Z info cache returned workload trust anchor from cache ttl=23h59m58.941585555s 2026-10-31T04:32:09.138895Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:32:17.339690Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:32:18.316147Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-10-31T04:32:19.506600Z info Readiness succeeded in 10.683635793s 2026-10-31T04:32:19.507036Z info Envoy proxy is ready 2026-10-31T04:32:25.507869Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:32:34.700663Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:32:37.841516Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-10-31T04:32:37.751Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 90 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "a22414cb-a8a6-4fb9-9abc-341d8b228453" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.29:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:48672 10.132.1.122:443 10.132.0.15:47104 - v1-catch-all-path 2026-10-31T04:32:51.567774Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:33:03.347423Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:33:07.930740Z info ads ADS: new connection for node:3 2026-10-31T04:33:07.931197Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-31T04:33:07.931290Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-31T04:33:14.913852Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:33:54.218199Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T04:34:00.778Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 333 87 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "52e45b5b-c25e-4344-bc71-10e6897b584a" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.202:8000" outbound|54321||e2e-prio-pool-7010fecc-inference-pool-ip-529cbf0f.llm.svc.cluster.local 10.132.1.122:34780 10.132.1.122:443 10.132.0.15:41238 - v1-model-routing [2026-10-31T04:34:16.131Z] "POST /v1/chat/completions HTTP/1.1" 403 - - "-" 131 0 35 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "0438b85d-7e44-4f1c-aaf5-8873d6326c51" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "-" outbound|54321||e2e-prio-pool-7010fecc-inference-pool-ip-529cbf0f.llm.svc.cluster.local - 10.132.1.122:443 10.132.0.15:55212 - v1-model-routing 2026-10-31T04:34:18.309849Z info ads ADS: "" 3 terminated 2026-10-31T04:34:18.620162Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:34:23.479426Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:34:40.485286Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:36:54.510979Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:36:54.511013Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:36:54.511024Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T04:36:54.511029Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T04:36:54.450Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 60 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "59d691e1-3aeb-4db8-81aa-60d0d46b6f93" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.22:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.122:37108 10.132.1.122:443 10.132.0.15:43310 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T04:37:11.170294Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:37:33.908773Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:38:01.297606Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:38:12.894881Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:38:19.591626Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:38:32.637656Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:38:37.439088Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:38:55.359243Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T04:39:13.189Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 283 84 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "1629e355-9c0b-4b47-adba-1aa7e2026a9b" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:40980 10.132.1.122:443 10.132.0.15:40392 - v1-completions-path 2026-10-31T04:39:43.507274Z info ads ADS: new connection for node:4 2026-10-31T04:39:43.515265Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-31T04:39:43.515354Z info cache read certificate from file resource=file-root:system 2026-10-31T04:39:47.449235Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:39:48.460318Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:39:51.645481Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"152","StreamId":"14428575445844686341"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-10-31T04:39:51.581Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 63 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "b6763a8f-04b3-484c-811b-303d91288c99" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.1.122:443 10.132.0.15:55910 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-31T04:40:27.309798Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:40:27.309836Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:40:27.309850Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:40:27.309856Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T04:40:27.310055Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T04:40:27.105Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1711 204 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "dbe45869-4899-44d4-8115-20e1e267c0b8" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.122:60906 10.132.1.122:443 10.132.0.15:53558 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-31T04:40:36.768960Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-31T04:40:36.769512Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:40:36.769558Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:40:36.769573Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:40:36.769577Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T04:40:36.769743Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T04:40:36.598Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1708 171 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "8e2a837a-e7e1-4d58-b539-d892e5507246" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.122:60906 10.132.1.122:443 10.132.0.15:44784 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-31T04:40:43.935988Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:40:43.936019Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:40:43.936029Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T04:40:43.936033Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:40:43.936188Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T04:40:43.776Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1702 159 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "2e3d0e55-55f0-43f0-95a2-ade80971c968" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.122:60906 10.132.1.122:443 10.132.0.15:41078 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-31T04:41:43.976606Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:41:44.003830Z info ads ADS: "" 4 terminated 2026-10-31T04:41:46.709199Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:41:57.099686Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:42:00.722687Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-31T04:42:00.644Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 79 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "f0add5e4-b178-4bca-a7cf-9dde13d7074d" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:57526 10.132.1.122:443 10.132.0.15:57068 - v1-completions-path 2026-10-31T04:42:16.370600Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:42:20.934440Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T04:42:25.831Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 60 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "5e5cd200-d5db-4c08-9cc5-fd352295aeb4" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:57526 10.132.1.122:443 10.132.0.15:42394 - v1-completions-path 2026-10-31T04:42:27.966753Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:42:27.966790Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:42:27.966805Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:42:27.966810Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-31T04:42:27.903Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 170 63 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "c6387986-e4a9-4c31-aac4-be7d5aa111ba" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:37878 10.132.1.122:443 10.132.0.15:42412 - v1-catch-all-path 2026-10-31T04:42:45.033383Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:42:59.783314Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T04:43:11.355Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 64 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "06a29ca2-92f8-475f-9049-a06855a98fb6" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:37878 10.132.1.122:443 10.132.0.15:38596 - v1-completions-path [2026-10-31T04:43:11.629Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "8b327cb0-0198-47f1-9a00-739fbb74d955" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:57526 10.132.1.122:443 10.132.0.15:38606 - v1-completions-path [2026-10-31T04:43:12.147Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 61 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "68ffb924-c991-4f0f-be49-e8bbd2963074" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:57526 10.132.1.122:443 10.132.0.15:38618 - v1-completions-path [2026-10-31T04:43:12.893Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "a20a6bed-ce6a-49ee-961f-c4f137acea23" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:57526 10.132.1.122:443 10.132.0.15:38644 - v1-completions-path 2026-10-31T04:43:31.971222Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:43:42.627083Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:43:48.978883Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:44:16.985873Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:44:21.357177Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:44:32.701651Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:44:41.178553Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:44:51.345063Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:44:59.820366Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:45:08.932560Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:45:27.064545Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:45:33.036551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T04:45:38.059Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 69 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "0842bc41-9005-47b0-9b4d-2b9a0975f146" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:40980 10.132.1.122:443 10.132.0.15:52362 - v1-completions-path [2026-10-31T04:45:49.267Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 40 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "03f6f0a2-f6b7-463a-8435-4622a02877ef" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.122:443 10.132.0.15:35750 - v1-completions-path 2026-10-31T04:45:53.347974Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:45:58.377358Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:46:10.678373Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T04:46:19.692Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 61 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "a2b19268-40ce-43f8-951f-886fc1d95f2b" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:57526 10.132.1.122:443 10.132.0.15:59126 - v1-completions-path 2026-10-31T04:46:33.310470Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:46:44.108957Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:47:03.073313Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:47:10.595569Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:49:36.285339Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:51:05.331734Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:51:22.643862Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:51:27.761695Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:51:41.276198Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:51:56.746324Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:52:00.397796Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:52:00.828933Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:52:00.828985Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:52:00.829007Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:52:00.829015Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T04:52:00.661Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 314 167 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "d9630c59-07d3-4908-b398-c7e857cdc255" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.219:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.122:56044 10.132.1.122:443 10.132.0.15:34836 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T04:52:29.686564Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:52:40.360888Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:52:46.556004Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:53:00.360597Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:53:00.360640Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:53:00.360661Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:53:00.360669Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T04:53:00.247Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 310 113 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "7b8a4e7b-bac5-4d82-9634-edd066ef2db8" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.132.1.152:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.122:56570 10.132.1.122:443 10.132.0.15:50158 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T04:53:00.821Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 62 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "e30288ea-aea2-488f-897f-bcae2c732130" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:57526 10.132.1.122:443 10.132.0.15:50162 - v1-completions-path [2026-10-31T04:53:01.451Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 61 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "74d4e232-9806-47ef-aed4-60c664645101" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:37878 10.132.1.122:443 10.132.0.15:50168 - v1-completions-path 2026-10-31T04:53:03.188303Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:53:03.188351Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:53:03.188374Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:53:03.188381Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T04:53:03.080Z] "DELETE /v1/api-keys/4653c7ab-6354-4d81-9623-296a37f5fc06 HTTP/1.1" 200 - via_upstream - "-" 0 365 108 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "3c9ff326-8ba1-4e31-bb84-8cf230a0c681" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.219:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.122:50178 10.132.1.122:443 10.132.0.15:50184 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T04:53:16.909249Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:53:23.316038Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:53:29.797348Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T04:53:38.119Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 71 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "142f9488-fc63-4e14-90bf-c1ad1f859e4a" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "10.133.0.31:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.122:37878 10.132.1.122:443 10.132.0.15:55740 - v1-completions-path [2026-10-31T04:53:44.347Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 61 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "3eef3ae8-a5d4-4695-8461-aab202cc994e" "maas.apps.6b93ef1713d6ca59dc0c.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.122:443 10.132.0.15:46814 - v1-completions-path 2026-10-31T04:53:58.706468Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:54:09.356128Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:54:16.129167Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:54:22.330931Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:54:32.083187Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:54:57.181750Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:55:07.961492Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:55:12.906133Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:55:30.929161Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:55:37.443490Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:55:58.530842Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:56:18.405992Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:56:29.391440Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:56:40.243230Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:56:51.313016Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:57:01.967361Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:57:12.903864Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:57:23.465818Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:57:34.282845Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:57:45.047594Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:57:55.780455Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8