2026-11-02T23:02:28.445587Z info FLAG: --concurrency="0" 2026-11-02T23:02:28.445606Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T23:02:28.445610Z info FLAG: --help="false" 2026-11-02T23:02:28.445613Z info FLAG: --log_as_json="false" 2026-11-02T23:02:28.445615Z info FLAG: --log_caller="" 2026-11-02T23:02:28.445617Z info FLAG: --log_output_level="default:info" 2026-11-02T23:02:28.445619Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T23:02:28.445624Z info FLAG: --log_target="[stdout]" 2026-11-02T23:02:28.445627Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T23:02:28.445629Z info FLAG: --outlierLogPath="" 2026-11-02T23:02:28.445631Z info FLAG: --profiling="true" 2026-11-02T23:02:28.445633Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T23:02:28.445635Z info FLAG: --proxyLogLevel="warning" 2026-11-02T23:02:28.445637Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T23:02:28.445639Z info FLAG: --stsPort="0" 2026-11-02T23:02:28.445641Z info FLAG: --templateFile="" 2026-11-02T23:02:28.445643Z info FLAG: --tokenManagerPlugin="" 2026-11-02T23:02:28.445646Z info FLAG: --vklog="0" 2026-11-02T23:02:28.445648Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T23:02:28.445788Z info Proxy role ips=[10.134.0.39] type=router id=maas-default-gateway-openshift-default-77bfb58f8c-2nshh.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T23:02:28.445831Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T23:02:28.446918Z info cpu limit detected as 2, setting concurrency 2026-11-02T23:02:28.447122Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T23:02:28.447133Z info JWT policy is third-party-jwt 2026-11-02T23:02:28.447136Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T23:02:28.448787Z info Starting default Istio SDS Server 2026-11-02T23:02:28.448810Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T23:02:28.448827Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T23:02:28.448820Z info Opening status port 15020 2026-11-02T23:02:28.449577Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T23:02:28.451216Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T23:02:28.452784Z info Starting proxy agent 2026-11-02T23:02:28.452816Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T23:02:28.471138Z info sds Starting SDS grpc server 2026-11-02T23:02:28.471191Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T23:02:28.519288Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T23:02:28.532291Z info cache generated new workload certificate resourceName=default latency=82.426109ms ttl=23h59m59.467713985s 2026-11-02T23:02:28.532332Z info cache Root cert has changed, start rotating root cert 2026-11-02T23:02:28.532374Z info cache returned workload trust anchor from cache ttl=23h59m59.467627263s 2026-11-02T23:02:28.667917Z info ads ADS: new connection for node:1 2026-11-02T23:02:28.667976Z info cache returned workload certificate from cache ttl=23h59m59.332026416s 2026-11-02T23:02:28.668183Z info ads ADS: new connection for node:2 2026-11-02T23:02:28.668255Z info cache returned workload trust anchor from cache ttl=23h59m59.331746639s 2026-11-02T23:02:28.751471Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:02:36.684959Z info Readiness succeeded in 8.241997668s 2026-11-02T23:02:36.685292Z info Envoy proxy is ready [2026-11-02T23:02:37.204Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 63 - "34.228.250.194,10.132.0.10" "curl/7.76.1" "c05a43c3-a338-432d-8f31-d7d7b2959fbb" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:47840 - v1-model-routing 2026-11-02T23:02:37.499781Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T23:02:37.499809Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T23:02:37.499812Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T23:02:37.499816Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-02T23:02:37.481Z] "POST /v1/chat/completions HTTP/1.1" 401 - - "-" 122 88 17 - "34.228.250.194,10.132.0.10" "curl/7.76.1" "74b2955f-11ba-4952-aac0-e0dff69fb422" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:47848 - v1-model-routing 2026-11-02T23:02:37.837531Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T23:02:37.837569Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T23:02:37.837584Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T23:02:37.837588Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T23:02:38.372395Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:02:38.372433Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:02:38.372446Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T23:02:38.372451Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:02:38.671684Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:02:38.671732Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:02:38.671755Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:02:38.671762Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T23:02:37.715Z] "DELETE /v1/api-keys/b41fe27a-730a-41e2-9ebe-88f7fb27a234 HTTP/1.1" 200 - via_upstream - "-" 0 365 121 - "34.228.250.194,10.132.0.10" "curl/7.76.1" "790610be-ccff-437d-89dd-c8cacb90b785" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:56338 10.134.0.39:443 10.132.0.10:43884 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T23:02:38.358Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 13 - "34.228.250.194,10.132.0.10" "curl/7.76.1" "7aadc8f9-b48b-4b41-a995-fd52041bef86" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:56344 10.134.0.39:443 10.132.0.10:43900 - odh-ai-gateway-infra.maas-api-route.4 [2026-11-02T23:02:38.562Z] "GET /v1/subscriptions HTTP/1.1" 200 - via_upstream - "-" 0 488 109 - "34.228.250.194,10.132.0.10" "curl/7.76.1" "a72b76bd-c64b-49c2-bf3a-04439374e6c5" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:56344 10.134.0.39:443 10.132.0.10:43908 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T23:08:04.204830Z info ads ADS: new connection for node:3 2026-11-02T23:08:04.210284Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T23:08:04.210334Z info cache read certificate from file resource=file-root:system 2026-11-02T23:08:09.690501Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:08:21.658581Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:08:38.171387Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:08:45.416396Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:08:45.416432Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:08:45.416441Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T23:08:45.416446Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:08:45.416597Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T23:08:45.216Z] "POST /anything/e2e-rt-sw-0096c8-client HTTP/1.1" 200 - via_upstream - "-" 124 1767 200 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "a5ea72b1-5b5c-4eaa-ae14-bce9677fba29" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.39:46956 10.134.0.39:443 10.132.0.10:41036 - llm.external-model-e2e-rt-sw-0096c8-f68826799e61.0 2026-11-02T23:08:46.019626Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:08:46.019660Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:08:46.019670Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:08:46.019675Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T23:08:46.019838Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-11-02T23:08:46.264106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:08:46.264138Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:08:46.264179Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T23:08:46.264186Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:08:46.264346Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T23:08:45.827Z] "POST /anything/e2e-rt-sw-0096c8-client HTTP/1.1" 200 - via_upstream - "-" 124 1769 192 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "8c5e5ab9-d0c6-4a2d-a37d-8331ebd6ccab" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.39:46960 10.134.0.39:443 10.132.0.10:41056 - llm.external-model-e2e-rt-sw-0096c8-f68826799e61.0 [2026-11-02T23:08:46.205Z] "POST /llm/e2e-rt-sw-0096c8/anything/e2e-rt-sw-0096c8-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 58 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "e7350461-8df1-4fda-807c-09fdd20d274b" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-0096c8.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:41072 - llm.external-model-e2e-rt-sw-0096c8-f68826799e61.2 2026-11-02T23:09:00.895349Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:09:25.726516Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-02T23:09:25.730207Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:09:25.730229Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:09:25.730240Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:09:25.730245Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T23:09:25.730412Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T23:09:25.661Z] "POST /llm/e2e-rt-sw-0096c8/anything/e2e-rt-sw-0096c8-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 69 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "50814ada-4911-48d1-86c8-6aba207710d6" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-0096c8.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:59750 - llm.external-model-e2e-rt-sw-0096c8-f68826799e61.2 2026-11-02T23:10:02.690165Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:10:20.615694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T23:11:54.614Z] "POST /llm/e2e-rt-eq-6fcabd/anything/e2e-rt-eq-6fcabd-client HTTP/1.1" 503 - - "-" 124 0 72 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "5b323fd1-66b2-4668-a99e-2fcaf0f53c58" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-eq-6fcabd.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:48498 - llm.external-model-e2e-rt-eq-6fcabd-1f470821c94d.4 [2026-11-02T23:12:39.526Z] "POST /llm/e2e-rt-eq-6fcabd/anything/e2e-rt-eq-6fcabd-client HTTP/1.1" 503 - - "-" 124 0 66 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "0b4d0eec-5c89-4d7e-a69f-e2de3a7e2c92" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-eq-6fcabd.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:40154 - llm.external-model-e2e-rt-eq-6fcabd-1f470821c94d.4 2026-11-02T23:13:20.147800Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:13:46.800560Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:14:04.548274Z info ads ADS: "" 3 terminated 2026-11-02T23:14:06.472307Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:14:19.945111Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:15:21.039575Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:16:24.026065Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T23:18:20.902Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 83 0 56 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "cd338b68-c5c8-4c27-9236-4dd7a8fb4d13" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.39:443 10.132.0.10:55464 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T23:18:22.151697Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T23:18:22.151734Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T23:18:22.151748Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T23:18:22.151753Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T23:18:22.038Z] "DELETE /v1/api-keys/e56b5e57-2689-4eda-beb8-d5ecbe2aa6ef HTTP/1.1" 200 - via_upstream - "-" 0 321 113 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "8d9a5183-b43c-4f40-80b6-1dfb8f03047b" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:35218 10.134.0.39:443 10.132.0.10:55468 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T23:18:22.432Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 88 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "3bcf339c-9155-4a74-a3af-4780018bd59a" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.39:443 10.132.0.10:55470 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T23:18:25.657139Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T23:18:25.657201Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T23:18:25.657222Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T23:18:25.657230Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T23:18:25.542Z] "DELETE /v1/api-keys/bde8aa7b-4008-4cf7-ac6c-1f826ef48dba HTTP/1.1" 200 - via_upstream - "-" 0 321 114 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "47b1c9f3-75dc-4a20-9e26-b87ff1be35d1" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:35218 10.134.0.39:443 10.132.0.10:55482 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T23:18:47.273550Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T23:18:50.042Z] "POST /maas-api/internal/v1/api-keys/validate HTTP/1.1" 404 NR route_not_found - "-" 31 0 18 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "0ca530fb-510e-45c4-921e-191a52a40880" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" - - 10.134.0.39:443 10.132.0.10:54550 - - 2026-11-02T23:19:22.085377Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T23:32:30.947Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 135 2829 80 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "db53d874-54a5-44a1-a536-91300bc295c9" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:57828 10.134.0.39:443 10.132.0.10:37832 - v1-chat-completions-path [2026-11-02T23:34:45.455Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 110 316 86 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "ae266898-b4ca-4276-bd16-c86471360e5f" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:57828 10.134.0.39:443 10.132.0.10:58752 - v1-chat-completions-path [2026-11-02T23:34:45.747Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 110 320 67 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "2479f2cd-ca44-455e-b039-ece4c4445434" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:44924 10.134.0.39:443 10.132.0.10:58764 - v1-chat-completions-path 2026-11-02T23:34:49.134784Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:34:49.134821Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:34:49.134834Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:34:49.134839Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T23:34:49.134992Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-11-02T23:34:49.135047Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T23:34:49.135055Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T23:34:49.135058Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T23:34:49.135062Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T23:34:49.068Z] "POST /v1/chat/completions HTTP/1.1" 404 - via_upstream - "-" 120 126 66 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "a0375374-2bc2-4139-a496-4b688c7cbf15" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:57828 10.134.0.39:443 10.132.0.10:56016 - v1-chat-completions-path 2026-11-02T23:35:21.361171Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 [2026-11-02T23:49:09.695Z] "GET /v1/tenants HTTP/1.1" 404 NR route_not_found - "-" 0 0 9 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "d8392198-f94a-4e17-a648-4befbdd19886" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" - - 10.134.0.39:443 10.132.0.10:57072 - - 2026-11-02T23:57:02.852052Z info ads ADS: new connection for node:4 2026-11-02T23:57:02.859133Z info cache read certificate from file resource=file-root:system 2026-11-02T23:57:07.437598Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:57:31.087137Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:57:43.102801Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:57:43.102836Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:57:43.102850Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:57:43.102855Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T23:57:42.914Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 66 308 188 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "be42923c-baec-4bb6-8731-baaae2a08fd6" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:53814 10.134.0.39:443 10.132.0.10:49648 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T23:57:43.395714Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-02T23:57:43.545351Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T23:57:43.545383Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T23:57:43.545395Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T23:57:43.545400Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T23:57:43.545561Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T23:57:43.313Z] "POST /anything/e2e-dft-647714-d1995c HTTP/1.1" 200 - via_upstream - "-" 109 1777 232 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "48c40be4-02e3-4b00-94c4-4e17f38096bd" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.39:40144 10.134.0.39:443 10.132.0.10:49652 - llm.external-model-e2e-dft-647714-82f8e8d2663b.0 2026-11-02T23:58:17.567428Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:59:14.798255Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:59:24.369573Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:59:24.369609Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:59:24.369623Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T23:59:24.369628Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T23:59:24.194Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 66 309 175 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "6584ee55-6219-4368-9e9f-6605b3c8d612" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:33854 10.134.0.39:443 10.132.0.10:58002 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:00:00.249837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:00:38.605Z] "POST /llm/e2e-dft-66da83/anything/e2e-dft-66da83-cbafab HTTP/1.1" 503 - - "-" 109 0 93 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "0ec66815-efd2-4812-8aa4-e9a7c1751cf7" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-66da83.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:40876 - llm.external-model-e2e-dft-66da83-99ca68864c21.2 [2026-11-03T00:00:43.903Z] "POST /llm/e2e-dft-66da83/anything/e2e-dft-66da83-cbafab HTTP/1.1" 503 - - "-" 109 0 64 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "03012177-a7cb-474a-9b43-64b985380e4b" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-66da83.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:40886 - llm.external-model-e2e-dft-66da83-99ca68864c21.2 [2026-11-03T00:01:26.366Z] "POST /llm/e2e-dft-66da83/anything/e2e-dft-66da83-cbafab HTTP/1.1" 503 - - "-" 109 0 71 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "bd3d5dcc-8891-4ece-abd9-d1407858aed5" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-66da83.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:44848 - llm.external-model-e2e-dft-66da83-99ca68864c21.2 [2026-11-03T00:02:24.805Z] "POST /llm/e2e-dft-66da83/anything/e2e-dft-66da83-cbafab HTTP/1.1" 503 - - "-" 109 0 69 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "abcffbb5-8284-40e2-99dd-6ef86ca48f7d" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-66da83.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:60720 - llm.external-model-e2e-dft-66da83-99ca68864c21.2 [2026-11-03T00:02:30.071Z] "POST /llm/e2e-dft-66da83/anything/e2e-dft-66da83-cbafab HTTP/1.1" 503 - - "-" 109 0 80 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "8b206605-1978-4c99-96b6-c37b00e917b8" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-66da83.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:38428 - llm.external-model-e2e-dft-66da83-99ca68864c21.2 [2026-11-03T00:02:46.067Z] "POST /llm/e2e-dft-66da83/anything/e2e-dft-66da83-cbafab HTTP/1.1" 503 - - "-" 109 0 84 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "e6306c36-c656-4dba-821e-cc4090778dea" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-66da83.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:59472 - llm.external-model-e2e-dft-66da83-99ca68864c21.2 [2026-11-03T00:03:07.132Z] "POST /llm/e2e-dft-66da83/anything/e2e-dft-66da83-cbafab HTTP/1.1" 503 - - "-" 109 0 67 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "1484ffe8-f355-4c02-bcc3-00bb8cdaaf23" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-66da83.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:36464 - llm.external-model-e2e-dft-66da83-99ca68864c21.2 2026-11-03T00:04:53.805892Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:04:56.967921Z info ads ADS: "" 4 terminated with stream closed 2026-11-03T00:04:57.107573Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:04:59.235036Z info ads ADS: new connection for node:5 2026-11-03T00:04:59.241473Z info cache read certificate from file resource=file-root:system 2026-11-03T00:05:03.488389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:05:04.786355Z info ads ADS: "" 5 terminated 2026-11-03T00:05:05.914833Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:05:11.307607Z info ads ADS: new connection for node:6 2026-11-03T00:05:11.328082Z info cache read certificate from file resource=file-root:system 2026-11-03T00:05:14.259101Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:05:17.637872Z info ads ADS: "" 6 terminated with stream closed 2026-11-03T00:05:18.276544Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:05:24.756566Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:05:57.524527Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:06:49.451368Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-03T00:06:58.811694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:07:28.904852Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:07:45.058722Z info ads ADS: new connection for node:7 2026-11-03T00:07:45.065551Z info cache read certificate from file resource=file-root:system 2026-11-03T00:07:49.605844Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:07:51.253558Z info ads ADS: "" 7 terminated with stream closed 2026-11-03T00:07:53.097044Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:08:10.550073Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:08:53.034991Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:21:47.433474Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:22:13.212294Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:22:22.207946Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:22:22.207979Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:22:22.207991Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:22:22.207996Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T00:22:22.094Z] "POST /v1/api-keys HTTP/1.1" 400 - via_upstream - "-" 80 94 113 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "a105bb08-7b05-407c-8444-42ae0eaab0a7" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:58500 10.134.0.39:443 10.132.0.10:46980 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:22:47.643674Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:22:50.576Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 2817 3089 76 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "0e2d9ad0-9d03-41be-afa2-c18c639a8bd8" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.38:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:41232 10.134.0.39:443 10.132.0.10:59046 - v1-chat-completions-path [2026-11-03T00:22:51.833Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 108118 108394 328 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "1a4f6ff7-bdb1-441c-ae8d-a1ec143948f8" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.38:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:38650 10.134.0.39:443 10.132.0.10:59062 - v1-chat-completions-path 2026-11-03T00:23:09.736186Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:23:12.450110Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T00:23:12.375Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 74 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "3ba8ba8b-28e3-466a-a3ea-d9b6ecc278de" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:51892 10.134.0.39:443 10.132.0.10:51508 - v1-catch-all-path 2026-11-03T00:23:17.579198Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:23:20.234004Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:23:20.234048Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:23:20.234067Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T00:23:20.234075Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:23:20.610079Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T00:23:20.068Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 318 165 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "a01c2dd3-0084-4530-a974-d1ecb1fdd36c" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:58500 10.134.0.39:443 10.132.0.10:55526 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T00:23:20.443Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 166 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "c4f223e9-613e-4ddb-b59f-414c24f38b4c" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:51892 10.134.0.39:443 10.132.0.10:55538 - v1-catch-all-path 2026-11-03T00:23:24.006874Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T00:23:23.871Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 135 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "3426f227-5f16-40a0-b5fa-a3984f516aaf" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:39174 10.134.0.39:443 10.132.0.10:55554 - v1-catch-all-path 2026-11-03T00:23:32.995384Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:23:42.976830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:23:59.749678Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:24:11.397794Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:24:14.785766Z info ads ADS: new connection for node:8 2026-11-03T00:24:14.785970Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T00:24:14.786015Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T00:24:18.622318Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:24:57.681336Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:25:01.163Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 333 101 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "32e9266e-900f-4175-be94-5ef1693b801c" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.217:8000" outbound|54321||e2e-prio-pool-78f9ed86-inference-pool-ip-18e30245.llm.svc.cluster.local 10.134.0.39:53870 10.134.0.39:443 10.132.0.10:56086 - v1-model-routing [2026-11-03T00:25:03.751Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 331 66 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "69dc4008-5e99-43f8-879a-03095efdb6e5" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.217:8000" outbound|54321||e2e-prio-pool-78f9ed86-inference-pool-ip-18e30245.llm.svc.cluster.local 10.134.0.39:53886 10.134.0.39:443 10.132.0.10:56110 - v1-model-routing 2026-11-03T00:25:22.016075Z info ads ADS: "" 8 terminated with stream closed 2026-11-03T00:25:22.452135Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:25:27.392009Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:25:43.317868Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:28:17.457443Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:28:43.633806Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:09.610114Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:23.508790Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:30.482791Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:41.149409Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:46.237973Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:49.199652Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:29:49.199692Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:29:49.199708Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T00:29:49.199712Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T00:29:49.142Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 524 56 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "03f8e321-1e78-4d8f-a90d-560a153f1db5" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:52052 10.134.0.39:443 10.132.0.10:47254 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:29:50.700887Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:29:50.700924Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:29:50.700939Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T00:29:50.700944Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T00:29:50.395Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 59 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "649a735b-8f5e-4b47-9887-59d2fd2fc5e0" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:47258 - v1-completions-path [2026-11-03T00:29:50.649Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 524 51 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "31497616-05e5-4c7a-bc6e-4e862250b8fa" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:52052 10.134.0.39:443 10.132.0.10:47268 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:30:04.638117Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:30:06.721863Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:30:06.721898Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:30:06.721912Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T00:30:06.721917Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T00:30:06.606Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 330 115 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "4d1bf7a6-56a8-4fa8-a720-9d343472efb5" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:52052 10.134.0.39:443 10.132.0.10:57150 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T00:30:21.452Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 66 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "9fcc0904-780f-4138-bcfe-3d81fec3a229" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:44924 10.134.0.39:443 10.132.0.10:56766 - v1-completions-path 2026-11-03T00:30:52.824945Z info ads ADS: new connection for node:9 2026-11-03T00:30:52.829113Z info cache read certificate from file resource=file-root:system 2026-11-03T00:30:56.548218Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:30:58.135241Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:31:03.428115Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"1270","StreamId":"8770969024363447668"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-03T00:31:03.363Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 64 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "1269a6cc-a22b-4772-b300-04183bdfda31" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.134.0.39:443 10.132.0.10:60314 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T00:31:10.365435Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:31:10.365474Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:31:10.365487Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:31:10.365493Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T00:31:10.365694Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T00:31:10.168Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1722 197 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "f2b664b8-5011-4d37-9946-dacf16699f85" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.39:35178 10.134.0.39:443 10.132.0.10:35168 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T00:31:22.302936Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:31:22.302968Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:31:22.302978Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T00:31:22.302982Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:31:22.303174Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T00:31:22.103Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1714 199 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "544153d0-a3b0-4f22-83fd-8a980a034e8b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.39:35592 10.134.0.39:443 10.132.0.10:48730 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T00:32:32.923631Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:32:32.923660Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:32:32.923670Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T00:32:32.923675Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:32:32.923813Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T00:32:32.805Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1721 118 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "d4f9cdff-6c16-406a-a8c4-ad207170c014" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.39:35178 10.134.0.39:443 10.132.0.10:39108 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T00:32:50.443166Z info ads ADS: "" 9 terminated with stream closed 2026-11-03T00:32:50.486269Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:32:53.310956Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:33:04.258968Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:33:22.982246Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:33:28.012171Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:33:33.475Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 59 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "48a9d342-94b8-4278-8cad-8797f52ba172" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:49042 - v1-completions-path 2026-11-03T00:33:47.825626Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:06.720750Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:34:13.406Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 59 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "1d668416-0931-4910-ad98-d47746c4aa41" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:36956 - v1-completions-path [2026-11-03T00:34:16.526Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 79 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "2f6a9992-6bca-48a0-a01c-18215b52088a" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:34862 10.134.0.39:443 10.132.0.10:36970 - v1-completions-path 2026-11-03T00:34:19.449989Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=4 2026-11-03T00:34:35.639142Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:46.446441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:51.246373Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:21.404094Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:26.296194Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:40.041565Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:49.302520Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:57.251924Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:36:06.730895Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:36:16.402458Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:36:35.060213Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:36:37.741930Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:36:37.741987Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:36:37.742012Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:36:37.742020Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T00:36:37.595Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 146 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "2f379b45-a83f-43be-b3b4-5dd8e07d212b" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:38704 10.134.0.39:443 10.132.0.10:60022 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:36:43.162188Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:36:53.556913Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:36:56.293Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 34 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "c715edb2-cbfa-463e-9a2e-06131d7551ca" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:48060 - v1-completions-path 2026-11-03T00:36:56.943689Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:36:56.943727Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:36:56.943742Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T00:36:56.943747Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T00:36:56.832Z] "DELETE /v1/api-keys/64b0d480-56c3-45c7-a540-a2ab11ae8f06 HTTP/1.1" 200 - via_upstream - "-" 0 375 111 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "ffbe1075-9db5-4c51-beaf-f4624fd924a3" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:38704 10.134.0.39:443 10.132.0.10:48066 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T00:36:58.264Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 0 32 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "b2647e34-65bb-4f0a-bb57-648009b91464" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.39:443 10.132.0.10:37874 - v1-completions-path 2026-11-03T00:37:06.472568Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:37:29.443970Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:37:40.040498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:38:59.802382Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-11-03T00:41:59.134407Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:42:16.329866Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:42:20.791846Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:42:33.356776Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:42:50.485972Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:42:55.137744Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:42:58.393199Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:42:58.393234Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:42:58.393250Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:42:58.393255Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T00:42:58.273Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 315 119 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "3bc6bdbc-b5a3-4957-894a-22bb8b59f33a" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.133.0.231:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:49104 10.134.0.39:443 10.132.0.10:32868 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:43:15.186325Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:43:25.744075Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:43:31.809504Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:43:59.007832Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:44:04.969782Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:44:10.661894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:44:18.085022Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:44:18.085060Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:44:18.085075Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:44:18.085080Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T00:44:17.968Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 68 310 116 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "42ca6d65-e248-4778-b212-6b4bec7e4742" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.248:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:33168 10.134.0.39:443 10.132.0.10:33676 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T00:44:21.269Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 65 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "d8befe29-4265-432e-8601-44dab9628a9b" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:34862 10.134.0.39:443 10.132.0.10:33694 - v1-completions-path [2026-11-03T00:44:24.182Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 69 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "2ad544c1-50d5-40eb-aac0-36872124eb70" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:34140 10.134.0.39:443 10.132.0.10:33698 - v1-completions-path 2026-11-03T00:44:27.869710Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:44:27.869754Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:44:27.869776Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T00:44:27.869783Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T00:44:27.752Z] "DELETE /v1/api-keys/b0c63d23-c05d-43ef-a736-cfd02fc0a721 HTTP/1.1" 200 - via_upstream - "-" 0 363 116 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "82f8e93a-a2a0-454a-a5d8-fbca3fc00d0d" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.248:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.39:33884 10.134.0.39:443 10.132.0.10:59128 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:44:41.811657Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:44:52.501021Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:44:58.909435Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:45:04.813260Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:45:17.040283Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:45:23.962Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 70 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "d43fccb4-3ff1-4d02-a045-d5e6c1797299" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:34140 10.134.0.39:443 10.132.0.10:60000 - v1-completions-path 2026-11-03T00:45:40.366091Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:45:51.370029Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:45:56.290667Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:46:13.609639Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:46:23.455779Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:46:44.001976Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:47:06.024007Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:47:28.078263Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:47:48.081016Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:47:59.081644Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:48:09.780792Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:48:20.524036Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:48:31.384243Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:48:42.041518Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:48:52.767Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 85 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "a54bc7d0-441f-422c-b89f-ed8b7263ba89" "maas.apps.9bf2ce48968e6843c10f.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.39:46454 10.134.0.39:443 10.132.0.10:33468 - v1-completions-path