2026-10-30T00:45:21.904458Z info FLAG: --concurrency="0" 2026-10-30T00:45:21.904477Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T00:45:21.904481Z info FLAG: --help="false" 2026-10-30T00:45:21.904484Z info FLAG: --log_as_json="false" 2026-10-30T00:45:21.904486Z info FLAG: --log_caller="" 2026-10-30T00:45:21.904488Z info FLAG: --log_output_level="default:info" 2026-10-30T00:45:21.904490Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T00:45:21.904546Z info FLAG: --log_target="[stdout]" 2026-10-30T00:45:21.904549Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T00:45:21.904551Z info FLAG: --outlierLogPath="" 2026-10-30T00:45:21.904553Z info FLAG: --profiling="true" 2026-10-30T00:45:21.904555Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T00:45:21.904557Z info FLAG: --proxyLogLevel="warning" 2026-10-30T00:45:21.904559Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T00:45:21.904561Z info FLAG: --stsPort="0" 2026-10-30T00:45:21.904562Z info FLAG: --templateFile="" 2026-10-30T00:45:21.904564Z info FLAG: --tokenManagerPlugin="" 2026-10-30T00:45:21.904567Z info FLAG: --vklog="0" 2026-10-30T00:45:21.904570Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T00:45:21.904735Z info Proxy role ips=[10.134.0.42] type=router id=maas-default-gateway-openshift-default-7bdc57c4cf-7nffr.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T00:45:21.904785Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T00:45:21.905993Z info cpu limit detected as 2, setting concurrency 2026-10-30T00:45:21.906233Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T00:45:21.906244Z info JWT policy is third-party-jwt 2026-10-30T00:45:21.906247Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T00:45:21.907759Z info Starting default Istio SDS Server 2026-10-30T00:45:21.907782Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T00:45:21.907805Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T00:45:21.908273Z info Opening status port 15020 2026-10-30T00:45:21.909006Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T00:45:21.909792Z info sds Starting SDS grpc server 2026-10-30T00:45:21.909821Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T00:45:21.910576Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T00:45:21.911399Z info Starting proxy agent 2026-10-30T00:45:21.911435Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T00:45:21.973820Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T00:45:22.049844Z info cache generated new workload certificate resourceName=default latency=139.947869ms ttl=23h59m59.950160284s 2026-10-30T00:45:22.049874Z info cache Root cert has changed, start rotating root cert 2026-10-30T00:45:22.049902Z info cache returned workload trust anchor from cache ttl=23h59m59.95009841s 2026-10-30T00:45:22.081678Z info ads ADS: new connection for node:1 2026-10-30T00:45:22.081761Z info cache returned workload certificate from cache ttl=23h59m59.918241602s 2026-10-30T00:45:22.081904Z info ads ADS: new connection for node:2 2026-10-30T00:45:22.081969Z info cache returned workload trust anchor from cache ttl=23h59m59.918032298s 2026-10-30T00:45:22.164811Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T00:45:30.104372Z info Readiness succeeded in 8.202093308s 2026-10-30T00:45:30.104689Z info Envoy proxy is ready 2026-10-30T00:45:40.867019Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:45:40.867072Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:45:40.867092Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:45:40.867100Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T00:45:40.732Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 55 321 134 - "44.212.242.249,10.132.0.19" "curl/7.76.1" "4eea327f-e598-4c5f-8b0d-b97f209d195f" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:52792 10.134.0.42:443 10.132.0.19:39794 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T00:45:41.819Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 56 - "44.212.242.249,10.132.0.19" "curl/7.76.1" "5c19d046-0883-423b-b479-8d605eeda6ec" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.42:443 10.132.0.19:39796 - v1-model-routing 2026-10-30T00:48:12.173067Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T00:48:12.173123Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T00:48:12.173143Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T00:48:12.173150Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T00:48:12.086Z] "DELETE /v1/api-keys/53843715-3d96-4fe4-826e-4a38105702e0 HTTP/1.1" 200 - via_upstream - "-" 0 370 86 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "6f473e95-3499-9d51-b090-c016906b18d8" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:53728 10.134.0.42:443 10.132.0.19:59328 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T00:48:13.144988Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:48:13.145021Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:48:13.145034Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:48:13.145039Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T00:48:12.548Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 44 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "4b683937-329b-43a9-bba3-a12e23ce8c3f" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.42:443 10.132.0.19:59332 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T00:48:13.075Z] "DELETE /v1/api-keys/8ff01c26-8cbf-43b5-bf96-7c22b69bc56b HTTP/1.1" 200 - via_upstream - "-" 0 321 69 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "ddf2bf63-8236-4ba6-be66-43e789641f67" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:53738 10.134.0.42:443 10.132.0.19:59336 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T00:48:13.661007Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:48:13.661039Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:48:13.661052Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:48:13.661057Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T00:48:13.589Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 71 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "b0ce2160-fdbd-4d30-b954-d0069772868b" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:53738 10.134.0.42:443 10.132.0.19:59342 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T00:48:14.335938Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T00:48:14.335970Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T00:48:14.335985Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T00:48:14.335989Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T00:48:14.268Z] "DELETE /v1/api-keys/50caef5f-0eb6-48cc-95ac-66de77f11ae8 HTTP/1.1" 200 - via_upstream - "-" 0 321 67 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "895d1d5d-4de8-498a-bc00-f84c0845a439" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:53728 10.134.0.42:443 10.132.0.19:59350 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T00:48:15.466636Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:48:15.466669Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:48:15.466682Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T00:48:15.466687Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T00:48:15.397Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 68 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "09bd3d3c-9b6c-4c5b-8d49-7c272a3e20ff" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:53738 10.134.0.42:443 10.132.0.19:59364 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T00:48:20.154116Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T00:48:20.154142Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T00:48:20.154146Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T00:48:20.154151Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-30T00:48:20.092Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 77 61 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "6746aa67-fa77-4109-bded-c4cc39e1ff72" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.42:443 10.132.0.19:45800 - v1-completions-path 2026-10-30T00:48:23.059941Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-10-30T00:48:23.062295Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:48:23.062326Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:48:23.062343Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T00:48:23.062348Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T00:48:22.620Z] "POST /maas-api/internal/v1/subscriptions/select HTTP/1.1" 404 NR route_not_found - "-" 56 0 10 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "49be2a7f-861a-4c3e-9137-d0800533e336" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "-" - - 10.134.0.42:443 10.132.0.19:45824 - - [2026-10-30T00:48:22.992Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 606 70 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "22d061d2-0a97-4ff5-b69e-3661a569feff" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:53738 10.134.0.42:443 10.132.0.19:45826 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T00:48:31.435794Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T00:48:45.259236Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T00:52:37.306452Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:52:37.306492Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T00:52:37.306509Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T00:52:37.306514Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T00:52:37.208Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 568 98 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "070e4b96-d7c5-4b8f-88d3-1c1946f5da4b" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:47104 10.134.0.42:443 10.132.0.19:44136 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T00:53:03.973932Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T00:53:35.153185Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T00:55:22.088497Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T00:57:53.648782Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T00:59:41.268362Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:01:10.477215Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:01:55.039562Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:14:15.816240Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T01:20:49.116910Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:20:49.116957Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:20:49.116975Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T01:20:49.116981Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:20:49.117233Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-10-30T01:20:49.117304Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T01:20:49.117315Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T01:20:49.117318Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T01:20:49.117323Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T01:20:49.034Z] "POST /v1/chat/completions HTTP/1.1" 404 - via_upstream - "-" 120 126 82 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "51e71dbb-d2f2-4727-94d1-c3e442a90e66" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:49396 10.134.0.42:443 10.132.0.19:36274 - v1-chat-completions-path 2026-10-30T01:26:38.083475Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:26:40.168405Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T01:26:40.168440Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T01:26:40.168454Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T01:26:40.168459Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T01:26:40.078Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 89 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "99910932-07f7-4b17-9ae8-9029a13ccc88" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:44884 10.134.0.42:443 10.132.0.19:44164 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T01:27:03.723107Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:27:44.434619Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T01:27:49.180Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 68 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "5e2d2e6f-8738-4f8d-8c13-99952c351a4f" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:44126 10.134.0.42:443 10.132.0.19:40480 - v1-completions-path 2026-10-30T01:28:01.606108Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:28:05.991477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T01:28:10.346Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 57 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "91fe1f61-8ac9-450a-85a8-24347d01d739" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:40948 10.134.0.42:443 10.132.0.19:41682 - v1-completions-path [2026-10-30T01:28:10.426Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 51 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "cdd8df5c-14df-41a5-9b35-425b51866c7c" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.42:443 10.132.0.19:41684 - v1-completions-path 2026-10-30T01:28:21.751793Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-30T01:28:21.671Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 279 80 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "78cb3541-6144-4140-80b7-14833af6b16a" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:37224 10.134.0.42:443 10.132.0.19:58738 - v1-completions-path 2026-10-30T01:28:24.306559Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:28:38.226252Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:28:43.181697Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-10-30T01:28:43.265294Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:28:43.265332Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:28:43.265346Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:28:43.265352Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T01:28:43.188Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 324 76 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "7949c4e6-7175-4afd-9477-fad3d0c9f352" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:39084 10.134.0.42:443 10.132.0.19:46716 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T01:28:44.254055Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-10-30T01:28:44.198Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 57 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "f52cb7c4-7ce4-4103-9fd1-38e23bbd84e5" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:40948 10.134.0.42:443 10.132.0.19:46732 - v1-completions-path [2026-10-30T01:28:45.689Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 53 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "6e1e7165-6376-4a79-99ea-c51175be9f5f" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:40948 10.134.0.42:443 10.132.0.19:46742 - v1-completions-path 2026-10-30T01:29:01.829580Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:29:12.183056Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:29:16.507472Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T01:29:20.719Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 60 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "243f860f-612d-4fb4-a614-455f5fd5ba94" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:40948 10.134.0.42:443 10.132.0.19:46144 - v1-completions-path 2026-10-30T01:29:33.422257Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:29:35.731565Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:29:54.812125Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:29:59.015170Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:30:15.608374Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:30:31.638081Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:30:42.447838Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:30:45.376454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:31:04.489864Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:31:07.314868Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T01:31:25.964Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 36 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "8b084097-66a9-4598-ab5a-6f68d43373f0" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.42:443 10.132.0.19:42840 - v1-completions-path 2026-10-30T01:31:28.250079Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:31:31.003991Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:33:07.706799Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:33:10.981180Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:34:40.843821Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:34:56.027445Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:34:58.232239Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:34:58.678477Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:34:58.678528Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:34:58.678553Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:34:58.678562Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T01:34:58.578Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 65 308 99 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "d84573b3-6863-4c9e-8dea-dd70859bdcbd" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:36698 10.134.0.42:443 10.132.0.19:50152 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T01:35:11.024806Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:36:29.540081Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:36:46.152739Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:37:01.016087Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:37:01.200960Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T01:37:01.201013Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T01:37:01.201032Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T01:37:01.201037Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T01:37:01.127Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 630 74 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "fede7080-7788-4e22-8833-a805e313908a" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:46230 10.134.0.42:443 10.132.0.19:37320 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T01:37:15.097888Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:37:19.413220Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:37:19.413253Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:37:19.413267Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T01:37:19.413272Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T01:37:19.535126Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T01:37:19.358Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 55 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "9e0747b9-c816-4828-a5d9-572db4fc138c" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.132.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:53414 10.134.0.42:443 10.132.0.19:33658 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T01:37:31.293612Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:37:36.445418Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T01:37:41.594Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "44.212.242.249,10.132.0.19" "python-requests/2.34.2" "a668a4d1-677e-4ac1-bc33-e36ea53cbdd8" "maas.apps.bea7b45522541d1e61da.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:44126 10.134.0.42:443 10.132.0.19:34344 - v1-completions-path 2026-10-30T01:37:53.766013Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:38:03.238135Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:38:09.051826Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:38:24.910243Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:38:27.109153Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:38:42.107821Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T01:38:52.882743Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8