2026-11-01T10:04:21.719141Z info FLAG: --concurrency="0" 2026-11-01T10:04:21.719163Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-01T10:04:21.719167Z info FLAG: --help="false" 2026-11-01T10:04:21.719170Z info FLAG: --log_as_json="false" 2026-11-01T10:04:21.719172Z info FLAG: --log_caller="" 2026-11-01T10:04:21.719174Z info FLAG: --log_output_level="default:info" 2026-11-01T10:04:21.719176Z info FLAG: --log_stacktrace_level="default:none" 2026-11-01T10:04:21.719186Z info FLAG: --log_target="[stdout]" 2026-11-01T10:04:21.719189Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-01T10:04:21.719191Z info FLAG: --outlierLogPath="" 2026-11-01T10:04:21.719193Z info FLAG: --profiling="true" 2026-11-01T10:04:21.719195Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-01T10:04:21.719197Z info FLAG: --proxyLogLevel="warning" 2026-11-01T10:04:21.719199Z info FLAG: --serviceCluster="istio-proxy" 2026-11-01T10:04:21.719201Z info FLAG: --stsPort="0" 2026-11-01T10:04:21.719203Z info FLAG: --templateFile="" 2026-11-01T10:04:21.719205Z info FLAG: --tokenManagerPlugin="" 2026-11-01T10:04:21.719207Z info FLAG: --vklog="0" 2026-11-01T10:04:21.719210Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-01T10:04:21.719385Z info Proxy role ips=[10.134.0.243] type=router id=maas-default-gateway-openshift-default-746ff8f49f-vmqkd.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-01T10:04:21.719436Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-01T10:04:21.720590Z info cpu limit detected as 2, setting concurrency 2026-11-01T10:04:21.720936Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-01T10:04:21.720949Z info JWT policy is third-party-jwt 2026-11-01T10:04:21.720952Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-01T10:04:21.722641Z info Starting default Istio SDS Server 2026-11-01T10:04:21.722667Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-01T10:04:21.722690Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-01T10:04:21.723124Z info Opening status port 15020 2026-11-01T10:04:21.724127Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-01T10:04:21.725969Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-01T10:04:21.727097Z info Starting proxy agent 2026-11-01T10:04:21.727142Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-01T10:04:21.734770Z info sds Starting SDS grpc server 2026-11-01T10:04:21.734792Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-01T10:04:21.796209Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-01T10:04:21.887074Z info cache generated new workload certificate resourceName=default latency=162.55577ms ttl=23h59m59.112931884s 2026-11-01T10:04:21.887119Z info cache Root cert has changed, start rotating root cert 2026-11-01T10:04:21.887160Z info cache returned workload trust anchor from cache ttl=23h59m59.112840447s 2026-11-01T10:04:21.942774Z info ads ADS: new connection for node:1 2026-11-01T10:04:21.942845Z info cache returned workload certificate from cache ttl=23h59m59.057157173s 2026-11-01T10:04:21.943101Z info ads ADS: new connection for node:2 2026-11-01T10:04:21.943177Z info cache returned workload trust anchor from cache ttl=23h59m59.05682358s 2026-11-01T10:04:22.026244Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:04:31.044083Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:04:31.988479Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-01T10:04:32.186851Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-01T10:04:33.187599Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-01T10:04:33.853446Z info Readiness succeeded in 12.137098085s 2026-11-01T10:04:33.853799Z info Envoy proxy is ready 2026-11-01T10:04:59.950092Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:05:19.293474Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:05:23.966492Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:05:33.935654Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:05:45.319985Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:05:55.033568Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:05:58.883854Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T10:05:58.718Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 165 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "8c5bc071-fa2d-4eaa-8614-9463f8616169" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:35890 10.134.0.243:443 10.133.0.18:57216 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T10:06:12.739594Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:06:24.386999Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:06:27.011833Z info ads ADS: new connection for node:3 2026-11-01T10:06:27.012127Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T10:06:27.012206Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T10:06:31.175977Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:07:10.242357Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T10:07:21.271Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 333 93 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "fb816421-c745-4223-9d1a-e9084c72e8d7" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.132.1.120:8000" outbound|54321||e2e-prio-pool-d609fa82-inference-pool-ip-949aac53.llm.svc.cluster.local 10.134.0.243:53422 10.134.0.243:443 10.133.0.18:52308 - v1-model-routing 2026-11-01T10:07:35.582609Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:07:35.611191Z info ads ADS: "" 3 terminated 2026-11-01T10:07:41.930378Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:07:54.497413Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:10:11.211232Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T10:10:11.633882Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T10:10:11.089Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 33 326 121 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "c2af9661-ed70-4e40-b885-2ea2c96b458e" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:34230 10.134.0.243:443 10.133.0.18:59930 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-01T10:10:11.565Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 68 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "e3269ee1-3446-4fce-81f2-a254d03b09d3" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:34230 10.134.0.243:443 10.133.0.18:59946 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T10:10:31.384438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:10:58.911656Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:11:29.065340Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:11:42.692456Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:11:48.686293Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:11:53.882106Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:12:06.813871Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:12:14.355149Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:12:18.411431Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T10:12:18.354Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 524 56 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "e8fee6ab-acab-42c7-9784-c0bb8c715ed4" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:53352 10.134.0.243:443 10.133.0.18:40610 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-01T10:12:19.166Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 61 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "f5fe0666-05fd-4ba3-824e-d5cfe75a791f" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.243:443 10.133.0.18:40618 - v1-completions-path 2026-11-01T10:12:33.128396Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:13:26.485838Z info ads ADS: new connection for node:4 2026-11-01T10:13:26.489198Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-01T10:13:26.489241Z info cache read certificate from file resource=file-root:system 2026-11-01T10:13:30.453970Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:13:31.219645Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:13:56.223175Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-01T10:13:56.223449Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-11-01T10:13:56.025Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1721 198 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "d7735adb-d9e4-4331-ab7a-fb0e033d3540" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.243:38120 10.134.0.243:443 10.133.0.18:52432 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-01T10:13:58.661477Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T10:13:58.661712Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-01T10:13:58.464Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 197 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "9a9f63de-016e-4d3f-b897-d4b7ec3700e0" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.243:38136 10.134.0.243:443 10.133.0.18:52434 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-01T10:14:11.755912Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T10:14:11.756205Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-01T10:14:08.593Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1720 3162 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "c937fa40-4432-452e-bea9-822c67baf5b2" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.243:38136 10.134.0.243:443 10.133.0.18:50634 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-01T10:15:08.477760Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T10:15:08.478095Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-01T10:15:08.343Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 134 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "e12e762e-72a7-4119-81cc-d3824f606aeb" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.243:38136 10.134.0.243:443 10.133.0.18:52280 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-01T10:15:31.403262Z info ads ADS: "" 4 terminated with stream closed 2026-11-01T10:15:31.456813Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:15:33.913418Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:15:44.235178Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:15:47.937992Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T10:15:47.783Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 87 329 154 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "11b110ae-1a09-4cbe-b687-3f171549c5f4" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:37184 10.134.0.243:443 10.133.0.18:56784 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T10:16:03.480286Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:16:09.531755Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T10:16:13.667Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 84 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "5ad1f462-8407-4d6e-b924-f2433bd83fab" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.243:50738 10.134.0.243:443 10.133.0.18:53740 - v1-completions-path 2026-11-01T10:16:31.363828Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:16:44.780680Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:16:51.436092Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T10:16:51.322Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 113 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "d57e7d23-506a-4cdf-8160-53ad5b17ffba" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:50978 10.134.0.243:443 10.133.0.18:46312 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T10:16:57.937963Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-01T10:16:57.885Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 62 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "0e385902-2679-41a3-9ce5-2346cb4faec4" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.243:60988 10.134.0.243:443 10.133.0.18:56898 - v1-completions-path 2026-11-01T10:17:15.875192Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:17:26.508044Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:17:32.668013Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:17:43.654820Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T10:17:43.540Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 317 114 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "87f397f0-d203-4077-b4f5-7708c0cb528f" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:38208 10.134.0.243:443 10.133.0.18:47816 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T10:18:01.210158Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:18:05.759010Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:18:06.257772Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T10:18:06.113Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 144 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "f1e195df-20b8-4da8-a354-2e924683c247" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.52:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:44976 10.134.0.243:443 10.133.0.18:41466 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T10:18:18.594986Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:18:27.241326Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:18:36.329398Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:18:44.994685Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:18:54.034433Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:19:11.490824Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:19:17.299945Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T10:19:32.174Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 38 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "ea3cbfca-1803-4c99-a85b-038fff64390e" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.243:443 10.133.0.18:54780 - v1-completions-path 2026-11-01T10:19:36.191257Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:19:41.428806Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:19:53.878606Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:20:13.703317Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:20:24.401602Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:20:42.823063Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:20:50.280058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:22:55.489950Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:24:25.130451Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:24:45.538688Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:24:50.683584Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:25:04.277469Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:25:18.041445Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:25:23.214248Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:25:44.095668Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:25:54.755301Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:26:01.331457Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:26:14.472350Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T10:26:14.351Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 310 120 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "d43036f0-2873-4033-ac00-06f5f6110bfc" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.149:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:38334 10.134.0.243:443 10.133.0.18:54518 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T10:26:17.653904Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T10:26:17.541Z] "DELETE /v1/api-keys/31f9e17e-dd8a-48c7-a3bf-7fb99af6bff6 HTTP/1.1" 200 - via_upstream - "-" 0 365 112 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "6a233913-b139-43cb-99da-d4ed508e93e6" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.149:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:38348 10.134.0.243:443 10.133.0.18:44398 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T10:26:17.974387Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T10:26:17.855Z] "DELETE /v1/api-keys/aae01178-bcba-45c9-8850-3ed328c4e8eb HTTP/1.1" 200 - via_upstream - "-" 0 365 118 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "c69fbbb4-df2a-43f5-bc1e-ee91a4390053" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.134.1.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:57488 10.134.0.243:443 10.133.0.18:44400 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T10:26:31.515028Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:26:38.136878Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:26:44.305682Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:26:58.770631Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 [2026-11-01T10:26:58.683Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 91 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "735265da-44fc-4d9c-a727-0faabc52032a" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.243:50738 10.134.0.243:443 10.133.0.18:54152 - v1-completions-path 2026-11-01T10:27:16.246013Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:27:26.856250Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:27:31.928431Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:27:38.117705Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:27:49.499887Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T10:27:56.798Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 71 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "4e2a8dc9-817d-4eb2-b32b-64399bcceb42" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.243:50738 10.134.0.243:443 10.133.0.18:50678 - v1-completions-path [2026-11-01T10:27:57.576Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 69 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "28a80132-8a55-4ccf-98ae-4aa0ef7e1e69" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.243:60988 10.134.0.243:443 10.133.0.18:50692 - v1-completions-path [2026-11-01T10:27:57.946Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 64 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "ca2f20cc-f4a4-49bf-8ea3-bcbd64f2923d" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.243:60988 10.134.0.243:443 10.133.0.18:50706 - v1-completions-path [2026-11-01T10:27:58.667Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 74 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "614533f6-a64f-4536-8c0f-b9179dce073b" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.243:50738 10.134.0.243:443 10.133.0.18:50730 - v1-completions-path 2026-11-01T10:28:12.929019Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:28:23.694228Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:28:28.690427Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:28:46.636672Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:28:52.918301Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:29:16.926058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:29:36.925299Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:29:47.878275Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:29:58.857630Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:30:09.823995Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:30:20.576822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:30:31.534977Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:30:42.333498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:30:53.063176Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:31:03.857198Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:31:14.131969Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T10:31:14.001Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 330 130 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "0d5a6428-4ab2-479b-860e-d78961194a22" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "10.133.0.149:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.243:46210 10.134.0.243:443 10.133.0.18:48744 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T10:31:14.505561Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T10:31:21.720984Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-01T10:31:21.721013Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-01T10:31:21.721016Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-01T10:31:21.721020Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=26 [2026-11-01T10:31:21.694Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 26 - "52.71.36.72,10.133.0.18" "python-requests/2.34.2" "3b35c872-da74-4d4d-9014-4b1f3b5defc4" "maas.apps.97301875ced6cd69e07a.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.243:443 10.133.0.18:39508 - v1-completions-path