2026-11-03T06:03:26.642971Z info FLAG: --concurrency="0" 2026-11-03T06:03:26.642997Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T06:03:26.643001Z info FLAG: --help="false" 2026-11-03T06:03:26.643003Z info FLAG: --log_as_json="false" 2026-11-03T06:03:26.643005Z info FLAG: --log_caller="" 2026-11-03T06:03:26.643007Z info FLAG: --log_output_level="default:info" 2026-11-03T06:03:26.643009Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T06:03:26.643014Z info FLAG: --log_target="[stdout]" 2026-11-03T06:03:26.643018Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T06:03:26.643020Z info FLAG: --outlierLogPath="" 2026-11-03T06:03:26.643022Z info FLAG: --profiling="true" 2026-11-03T06:03:26.643024Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T06:03:26.643026Z info FLAG: --proxyLogLevel="warning" 2026-11-03T06:03:26.643028Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T06:03:26.643030Z info FLAG: --stsPort="0" 2026-11-03T06:03:26.643032Z info FLAG: --templateFile="" 2026-11-03T06:03:26.643033Z info FLAG: --tokenManagerPlugin="" 2026-11-03T06:03:26.643036Z info FLAG: --vklog="0" 2026-11-03T06:03:26.643039Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T06:03:26.643179Z info Proxy role ips=[10.133.0.37] type=router id=maas-default-gateway-openshift-default-778b7b5c47-ssx5p.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T06:03:26.643218Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T06:03:26.644316Z info cpu limit detected as 2, setting concurrency 2026-11-03T06:03:26.644525Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T06:03:26.644536Z info JWT policy is third-party-jwt 2026-11-03T06:03:26.644539Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T06:03:26.646157Z info Starting default Istio SDS Server 2026-11-03T06:03:26.646181Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T06:03:26.646204Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T06:03:26.646872Z info Opening status port 15020 2026-11-03T06:03:26.647302Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T06:03:26.648762Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T06:03:26.649574Z info Starting proxy agent 2026-11-03T06:03:26.649607Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T06:03:26.659300Z info sds Starting SDS grpc server 2026-11-03T06:03:26.659318Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T06:03:26.712096Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T06:03:26.822133Z info cache generated new workload certificate resourceName=default latency=175.004511ms ttl=23h59m59.17787242s 2026-11-03T06:03:26.822166Z info cache Root cert has changed, start rotating root cert 2026-11-03T06:03:26.822201Z info cache returned workload trust anchor from cache ttl=23h59m59.177799606s 2026-11-03T06:03:26.838940Z info ads ADS: new connection for node:1 2026-11-03T06:03:26.838998Z info cache returned workload certificate from cache ttl=23h59m59.161003778s 2026-11-03T06:03:26.839262Z info ads ADS: new connection for node:2 2026-11-03T06:03:26.839333Z info cache returned workload trust anchor from cache ttl=23h59m59.160667983s 2026-11-03T06:03:26.919191Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:03:34.745763Z info Readiness succeeded in 8.105013219s 2026-11-03T06:03:34.746045Z info Envoy proxy is ready 2026-11-03T06:03:51.809563Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:03:51.809597Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:03:51.809611Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:03:51.809616Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T06:03:51.725Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 84 - "44.212.242.249,10.133.0.12" "curl/7.76.1" "bbdc8ff3-1ba8-4468-bd52-0addc7208470" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:37850 10.133.0.37:443 10.133.0.12:60168 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T06:03:51.875Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 412 82 - "44.212.242.249,10.133.0.12" "curl/7.76.1" "7d5260d1-f106-472b-aab6-2d09d54dd2e8" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:54954 10.133.0.37:443 10.133.0.12:60170 - v1-model-routing [2026-11-03T06:03:52.673Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 55 - "44.212.242.249,10.133.0.12" "curl/7.76.1" "7512969c-7b8b-4983-b5e1-0e918a4afb23" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.133.0.12:60172 - v1-model-routing [2026-11-03T06:03:52.812Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 51 - "44.212.242.249,10.133.0.12" "curl/7.76.1" "e10bb345-ed53-4e73-9159-12c03b479501" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.133.0.12:60178 - v1-model-routing 2026-11-03T06:06:43.786577Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:43.786614Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:43.786625Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:43.786630Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T06:06:43.973106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T06:06:43.973157Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T06:06:43.973175Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T06:06:43.973182Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T06:06:43.700Z] "DELETE /v1/api-keys/88b14d55-c8b1-4cad-9590-697578616308 HTTP/1.1" 200 - via_upstream - "-" 0 321 85 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "bf9e3a5f-5579-41a6-b1fa-ff7c4b6ea8d8" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:38724 10.133.0.37:443 10.133.0.12:47308 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T06:06:43.891Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 82 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "96d94f34-82d7-4970-aa14-188df06bd996" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:38726 10.133.0.37:443 10.133.0.12:47320 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T06:06:44.163730Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:44.163777Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:44.163795Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:44.163803Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T06:06:44.465215Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:44.465272Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:44.465292Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T06:06:44.465299Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T06:06:44.093Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 312 70 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "573dc49c-cc8a-45a0-8081-636709892c76" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:38724 10.133.0.37:443 10.133.0.12:47336 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T06:06:44.385Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 79 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "beb2fb6d-f79b-41a0-a84a-80f6ba845d6f" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:38724 10.133.0.37:443 10.133.0.12:47346 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T06:06:44.612Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 60 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "b7f99e7b-580e-4d33-81b9-0e05b7f49c71" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.37:443 10.133.0.12:47350 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T06:06:45.166889Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:45.166922Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-03T06:06:45.166941Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:45.166961Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T06:06:45.166967Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:45.183914Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:45.183997Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:06:45.184028Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T06:06:45.184048Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T06:06:45.083Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 83 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "23fdf1cc-9a0d-41fd-b7a5-8d490ed77d48" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:38724 10.133.0.37:443 10.133.0.12:47352 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T06:06:45.099Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 84 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "71d1382f-ad12-46e1-8a31-286cc7fdc55a" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:38724 10.133.0.37:443 10.133.0.12:47354 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T06:06:48.027Z] "POST /maas-api/internal/v1/api-keys/validate HTTP/1.1" 404 NR route_not_found - "-" 31 0 11 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "9f128853-1818-4eec-8309-d5af0d75f4ce" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "-" - - 10.133.0.37:443 10.133.0.12:47374 - - [2026-11-03T06:06:50.382Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 47 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "f4ebdce6-0033-4849-a670-adb14af2576c" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.133.0.12:36266 - v1-completions-path [2026-11-03T06:06:50.795Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 66 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "c6d27708-fb4d-422e-b63a-c67bc5c353d0" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:51218 10.133.0.37:443 10.133.0.12:36278 - v1-completions-path 2026-11-03T06:07:03.011108Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:07:17.815729Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:11:50.594886Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:12:10.528800Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:13:01.725803Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T06:13:06.869Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 62 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "9b50900a-53b1-4133-86bb-8e258f915420" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.133.0.12:44804 - v1-completions-path 2026-11-03T06:13:19.001112Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:13:23.663159Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:13:42.172933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:13:44.633611Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:13:48.054973Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:13:58.037114Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:14:05.019495Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-03T06:14:04.724Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 73 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "d9e3d55f-d723-4271-8511-7289058aa262" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:60668 10.133.0.37:443 10.133.0.12:45216 - v1-completions-path [2026-11-03T06:14:04.892Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 58 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "051f7e69-acdc-998c-b31a-6a11bd4f4c6d" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:60668 10.133.0.37:443 10.133.0.12:45232 - v1-completions-path [2026-11-03T06:14:04.968Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 54 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "9443f639-61c7-4327-b1a4-9c047d896ec8" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:60668 10.133.0.37:443 10.133.0.12:45246 - v1-completions-path 2026-11-03T06:14:21.120100Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:14:31.519238Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:14:36.027000Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T06:14:43.258Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 84 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "d349aeda-1622-4c1d-b2dc-f9b9f407e7c3" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:57710 10.133.0.37:443 10.133.0.12:40424 - v1-completions-path [2026-11-03T06:14:44.190Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 52 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "e2e043c2-cf23-449f-b53e-476d65d0ddd9" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:57710 10.133.0.37:443 10.133.0.12:40446 - v1-completions-path 2026-11-03T06:14:56.233423Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:14:58.660613Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:15:17.765651Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T06:15:19.525Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 63 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "512d57c7-2a87-4cba-9830-47554fa9d492" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:54954 10.133.0.37:443 10.133.0.12:42448 - v1-completions-path 2026-11-03T06:15:22.376836Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:15:40.904379Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:15:43.272747Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:15:59.831569Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-03T06:15:59.766Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 65 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "9dcc5b11-0d94-4048-8779-2d77dafffb87" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:51218 10.133.0.37:443 10.133.0.12:51786 - v1-completions-path 2026-11-03T06:16:01.132209Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:16:11.830328Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:16:15.149269Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:16:36.340975Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:16:39.371296Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:16:53.553472Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:17:11.619026Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:17:22.030687Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:17:28.161359Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:17:28.161411Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:17:28.161426Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T06:17:28.161432Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T06:17:28.088Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 72 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "b0da4f73-f534-43cd-b7a6-e1ede3e7baf9" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:38136 10.133.0.37:443 10.133.0.12:39162 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T06:17:41.046508Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:17:44.781374Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T06:19:26.837Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 47 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "b140a4e1-cf69-46b2-a026-91fded3b004c" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.133.0.12:54580 - v1-completions-path 2026-11-03T06:19:29.141470Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:19:32.916962Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:21:03.089219Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:21:17.889917Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:21:20.192859Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:21:26.874662Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T06:21:26.874694Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T06:21:26.874706Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T06:21:26.874711Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T06:21:26.801Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 65 307 73 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "ac22906a-42b3-442a-87cf-4beff24dfaf1" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:49580 10.133.0.37:443 10.133.0.12:44850 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T06:21:39.141977Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:23:06.334208Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T06:23:06.334239Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T06:23:06.334251Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T06:23:06.334255Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T06:23:06.272Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 61 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "18e28a5f-c3f1-4392-ae4c-7613c26d8c7b" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:43864 10.133.0.37:443 10.133.0.12:45744 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T06:23:11.187733Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:23:27.831370Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:23:49.118777Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:24:03.333084Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:24:07.586946Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:24:18.441294Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:24:22.104086Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:24:41.995544Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:27:30.416134Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T06:27:30.416160Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T06:27:30.416163Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T06:27:30.416167Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-03T06:27:30.345Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 70 - "44.212.242.249,10.133.0.12" "python-requests/2.34.2" "a2d632cf-1cf3-473a-86b4-78d4d245c063" "maas.apps.a3c09cf78a357df5f249.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.133.0.12:45090 - v1-catch-all-path 2026-11-03T06:27:32.791022Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:27:41.251238Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:27:59.060404Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:28:01.153548Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:28:16.302538Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T06:28:27.160623Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7