2026-11-03T21:51:23.512015Z info FLAG: --concurrency="0" 2026-11-03T21:51:23.512035Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T21:51:23.512039Z info FLAG: --help="false" 2026-11-03T21:51:23.512041Z info FLAG: --log_as_json="false" 2026-11-03T21:51:23.512043Z info FLAG: --log_caller="" 2026-11-03T21:51:23.512045Z info FLAG: --log_output_level="default:info" 2026-11-03T21:51:23.512047Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T21:51:23.512052Z info FLAG: --log_target="[stdout]" 2026-11-03T21:51:23.512054Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T21:51:23.512056Z info FLAG: --outlierLogPath="" 2026-11-03T21:51:23.512058Z info FLAG: --profiling="true" 2026-11-03T21:51:23.512060Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T21:51:23.512062Z info FLAG: --proxyLogLevel="warning" 2026-11-03T21:51:23.512064Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T21:51:23.512066Z info FLAG: --stsPort="0" 2026-11-03T21:51:23.512068Z info FLAG: --templateFile="" 2026-11-03T21:51:23.512069Z info FLAG: --tokenManagerPlugin="" 2026-11-03T21:51:23.512072Z info FLAG: --vklog="0" 2026-11-03T21:51:23.512074Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T21:51:23.512213Z info Proxy role ips=[10.132.0.107] type=router id=maas-default-gateway-openshift-default-74854d8ccf-w6978.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T21:51:23.512276Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T21:51:23.513209Z info cpu limit detected as 2, setting concurrency 2026-11-03T21:51:23.513435Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T21:51:23.513448Z info JWT policy is third-party-jwt 2026-11-03T21:51:23.513451Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T21:51:23.514955Z info Starting default Istio SDS Server 2026-11-03T21:51:23.514975Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T21:51:23.514991Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T21:51:23.515334Z info Opening status port 15020 2026-11-03T21:51:23.516063Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T21:51:23.517098Z info sds Starting SDS grpc server 2026-11-03T21:51:23.517116Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T21:51:23.517441Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T21:51:23.518234Z info Starting proxy agent 2026-11-03T21:51:23.518295Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T21:51:23.583057Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T21:51:23.717522Z info cache generated new workload certificate resourceName=default latency=200.272668ms ttl=23h59m59.282482007s 2026-11-03T21:51:23.717557Z info cache Root cert has changed, start rotating root cert 2026-11-03T21:51:23.717596Z info cache returned workload trust anchor from cache ttl=23h59m59.282404149s 2026-11-03T21:51:23.779207Z info ads ADS: new connection for node:1 2026-11-03T21:51:23.784823Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T21:51:23.784872Z info cache read certificate from file resource=file-root:system 2026-11-03T21:51:23.809935Z info ads ADS: new connection for node:2 2026-11-03T21:51:23.809994Z info cache returned workload certificate from cache ttl=23h59m59.190007542s 2026-11-03T21:51:23.810849Z info ads ADS: new connection for node:3 2026-11-03T21:51:23.810915Z info cache returned workload trust anchor from cache ttl=23h59m59.189086442s 2026-11-03T21:51:23.917453Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:51:33.263922Z info Readiness succeeded in 9.754478491s 2026-11-03T21:51:33.264493Z info Envoy proxy is ready 2026-11-03T21:51:33.903065Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:51:44.958135Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:51:58.719547Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:52:11.341519Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T21:52:11.341861Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T21:52:11.166Z] "POST /anything/e2e-rt-eq-2de06c-client HTTP/1.1" 200 - via_upstream - "-" 124 1757 175 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "ebcb35bf-971d-45ac-9126-18623bbda250" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.107:38012 10.132.0.107:443 10.132.0.9:57926 - llm.external-model-e2e-rt-eq-2de06c-1230e2946409.2 2026-11-03T21:52:35.947237Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:52:40.363822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:52:52.705968Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:53:25.504367Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:53:42.609991Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:53:57.059933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:54:05.206943Z info ads ADS: "" 1 terminated with stream closed 2026-11-03T21:54:05.285241Z info ads ADS: new connection for node:4 2026-11-03T21:54:05.290578Z info cache read certificate from file resource=file-root:system 2026-11-03T21:54:06.058376Z info ads ADS: "" 4 terminated with stream closed 2026-11-03T21:54:20.188025Z info ads ADS: new connection for node:5 2026-11-03T21:54:20.194179Z info cache read certificate from file resource=file-root:system 2026-11-03T21:54:24.865409Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:54:34.639748Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:54:50.218168Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T21:54:50.218458Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T21:54:50.043Z] "POST /anything/e2e-dft-07ad0e-785cff HTTP/1.1" 200 - via_upstream - "-" 109 1766 175 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "4ab3f16d-6401-4dae-a5f4-5f413a3fe7df" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.107:50536 10.132.0.107:443 10.132.0.9:42128 - llm.external-model-e2e-dft-07ad0e-92a4f924761c.0 2026-11-03T21:55:02.586572Z info ads ADS: "" 5 terminated 2026-11-03T21:55:03.439752Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:55:10.080574Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:55:15.635926Z info ads ADS: new connection for node:6 2026-11-03T21:55:15.643122Z info cache read certificate from file resource=file-root:system 2026-11-03T21:55:20.332919Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:55:21.399475Z info ads ADS: "" 6 terminated with stream closed 2026-11-03T21:55:24.035898Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:55:28.133421Z info ads ADS: new connection for node:7 2026-11-03T21:55:28.142734Z info cache read certificate from file resource=file-root:system 2026-11-03T21:55:30.526625Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:55:34.406783Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:55:44.418312Z info ads ADS: "" 7 terminated with stream closed 2026-11-03T21:55:45.267348Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:55:47.012740Z info ads ADS: new connection for node:8 2026-11-03T21:55:47.021941Z info cache read certificate from file resource=file-root:system 2026-11-03T21:55:51.722155Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:55:53.298909Z info ads ADS: "" 8 terminated with stream closed 2026-11-03T21:55:55.294897Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:55:59.544742Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:56:07.965015Z info ads ADS: new connection for node:9 2026-11-03T21:56:07.970594Z info cache read certificate from file resource=file-root:system 2026-11-03T21:58:07.406765Z info ads ADS: "" 9 terminated with stream closed 2026-11-03T21:58:26.201126Z info ads ADS: new connection for node:10 2026-11-03T21:58:26.216579Z info cache read certificate from file resource=file-root:system 2026-11-03T21:59:48.438475Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-03T22:00:54.392968Z info ads ADS: new connection for node:11 2026-11-03T22:00:54.393174Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T22:00:54.393201Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T22:01:53.963738Z info ads ADS: "" 11 terminated 2026-11-03T22:02:53.218215Z info ads ADS: "" 10 terminated 2026-11-03T22:04:57.453511Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T22:04:57.232Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 30 314 221 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "d3bdf4d5-5202-450d-8198-22dc396b4c9a" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.107:59086 10.132.0.107:443 10.132.0.9:55076 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T22:05:04.880813Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:05:29.514918Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:05:39.608430Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T22:05:39.439Z] "POST /v1/api-keys HTTP/1.1" 400 - via_upstream - "-" 80 94 168 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "211879fe-00a8-41a7-b876-ebf562b3019f" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.107:59086 10.132.0.107:443 10.132.0.9:44602 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T22:06:05.210515Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T22:06:12.292Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 140 405 115 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "05db4cf6-4850-4a0b-bf44-79838be87678" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:34428 10.132.0.107:443 10.132.0.9:33484 - v1-chat-completions-path 2026-11-03T22:06:26.102513Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:06:39.075361Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:06:48.414227Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:07:07.861357Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:07:12.358498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:07:13.449790Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T22:07:13.195Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 254 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "0aa9e0ac-1f27-4f60-af83-8cfef19766bb" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.107:47972 10.132.0.107:443 10.132.0.9:46866 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T22:07:16.229120Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T22:07:15.929Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 300 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "f7fb7fd1-0c21-4107-8d85-5abe0835af65" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:57564 10.132.0.107:443 10.132.0.9:46880 - v1-catch-all-path 2026-11-03T22:07:29.939762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:07:41.625068Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:07:44.963750Z info ads ADS: new connection for node:12 2026-11-03T22:07:44.963912Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T22:07:48.513428Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:08:37.822473Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:09:13.485484Z info ads ADS: "" 12 terminated with stream closed 2026-11-03T22:09:13.534522Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:09:19.912609Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:09:34.582973Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:12:38.830925Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:13:10.345716Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:13:40.390820Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:13:55.864204Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:14:16.918507Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:14:39.502436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:14:53.642204Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:14:57.460737Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T22:14:57.130Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 93 336 329 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "a896f35f-21df-4f84-8752-090561d1333e" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.107:33890 10.132.0.107:443 10.132.0.9:39328 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T22:15:12.891182Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:17:22.087072Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T22:17:22.087325Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 2026-11-03T22:17:22.087396Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T22:17:22.087404Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T22:17:22.087407Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T22:17:22.087411Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T22:17:21.975Z] "POST /v1/chat/completions HTTP/1.1" 404 - via_upstream - "-" 120 126 112 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "c12ef8d4-4774-492d-b44e-f8a64c994c09" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:50612 10.132.0.107:443 10.132.0.9:49060 - v1-chat-completions-path 2026-11-03T22:20:21.504878Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T22:20:21.257Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 330 247 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "2cd8dbac-e28a-4062-a3ff-e597a69fe6de" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.107:34170 10.132.0.107:443 10.132.0.9:40038 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T22:20:37.075Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 283 106 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "caf0886f-e318-4adb-83ea-015ccfd03cf4" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:50612 10.132.0.107:443 10.132.0.9:36464 - v1-completions-path [2026-11-03T22:20:50.252Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 109 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "d38958d2-cc43-4d8f-82b0-c3aa5f3be987" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:50612 10.132.0.107:443 10.132.0.9:56164 - v1-completions-path 2026-11-03T22:21:07.613058Z info ads ADS: new connection for node:13 2026-11-03T22:21:07.616626Z info cache read certificate from file resource=file-root:system 2026-11-03T22:21:11.459494Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:21:12.594839Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:21:15.821504Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"518","StreamId":"11378505034509917231"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-03T22:21:15.809Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 12 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "78290666-4c00-4ac2-b6c9-a176a827fce7" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.0.107:443 10.132.0.9:59918 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T22:21:20.412633Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"523","StreamId":"8736878145448757451"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-03T22:21:20.310Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 102 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "be63c7af-309f-4460-b594-7f96b1e0c9a7" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.0.107:443 10.132.0.9:59942 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T22:21:22.712863Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"526","StreamId":"2860871871304218846"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-03T22:21:22.616Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 96 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "8802c841-ee52-4d0a-8069-730934160eeb" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.0.107:443 10.132.0.9:46562 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T22:21:25.090533Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T22:21:25.090841Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T22:21:25.091063Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T22:21:24.927Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1711 163 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "0e9f389d-a09f-4d64-9ce1-ef6134fafd57" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.107:56736 10.132.0.107:443 10.132.0.9:46590 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T22:21:27.293594Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-03T22:21:27.439244Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T22:21:27.439624Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T22:21:27.287Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 152 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "b0f4900b-1676-403c-8683-4be09b08fc1d" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.107:56752 10.132.0.107:443 10.132.0.9:46604 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T22:21:32.158198Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T22:21:32.158438Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T22:21:32.005Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 152 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "f7703db2-0233-483e-8136-e8777935de16" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.107:56736 10.132.0.107:443 10.132.0.9:46622 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T22:21:36.885485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T22:21:36.885730Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T22:21:36.726Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 159 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "8cb2503b-47b0-46e5-a6b0-5380cef37d3b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.107:56736 10.132.0.107:443 10.132.0.9:34210 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T22:21:49.427702Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T22:21:49.427964Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T22:21:49.273Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1705 154 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "fe200d4d-176d-46f3-86a5-fe5d7b6b7b3b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.107:56752 10.132.0.107:443 10.132.0.9:38418 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T22:22:06.652589Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T22:22:06.652830Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T22:22:06.502Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1711 150 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "cc7a4eba-a762-4971-adcf-31c89005049d" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.107:56736 10.132.0.107:443 10.132.0.9:55568 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T22:22:13.712075Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T22:22:13.712358Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T22:22:13.569Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1707 143 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "104d1079-4ebc-4c0e-af1b-3c2a555018ee" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.107:56736 10.132.0.107:443 10.132.0.9:58590 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T22:22:40.363371Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T22:22:40.363654Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T22:22:37.186Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1710 3176 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "8f01582d-7b52-4eb4-bc18-16eaa7bf0bc8" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.107:56752 10.132.0.107:443 10.132.0.9:44118 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T22:22:55.743423Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:22:55.762458Z info ads ADS: "" 13 terminated 2026-11-03T22:22:58.322888Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:23:00.708543Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T22:23:00.708568Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T22:23:00.708572Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T22:23:00.708576Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-03T22:23:00.624Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 83 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "38eb63c0-ffe1-461d-9379-4845f9cb588d" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.107:443 10.132.0.9:51508 - v1-completions-path 2026-11-03T22:23:08.886931Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:23:10.009200Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T22:23:09.801Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 96 338 207 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "f8b6ff42-d3a4-48cd-8919-84659a1eb691" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.107:54064 10.132.0.107:443 10.132.0.9:46834 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T22:23:28.313198Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:23:32.854415Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T22:23:37.369Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 117 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "99c3b61c-4951-4477-90aa-6570cf0c8b1c" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60730 10.132.0.107:443 10.132.0.9:58720 - v1-completions-path [2026-11-03T22:23:38.043Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 299 105 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "ac5f5cd3-208a-4b2d-bffa-c50ed3afcfbe" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60742 10.132.0.107:443 10.132.0.9:58748 - v1-completions-path [2026-11-03T22:23:38.346Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 288 103 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "9766ac6f-385c-44a8-9fc7-b2a4ab876647" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60742 10.132.0.107:443 10.132.0.9:58760 - v1-completions-path [2026-11-03T22:23:39.273Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 93 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "ee560df2-d449-4943-a5cb-1d85c6ff42a8" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.107:443 10.132.0.9:58772 - v1-completions-path 2026-11-03T22:23:56.643518Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:24:09.910202Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T22:24:16.959Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 99 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "2a02abc8-e2cc-4e9a-acc1-51a07dccef50" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60742 10.132.0.107:443 10.132.0.9:41062 - v1-completions-path [2026-11-03T22:24:17.671Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 94 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "a9557f4d-ceb0-4cda-b3d2-4689fda9b160" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60730 10.132.0.107:443 10.132.0.9:41064 - v1-completions-path [2026-11-03T22:24:22.297Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 97 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "058783da-207e-4b96-931b-6c2cca88e9c1" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60730 10.132.0.107:443 10.132.0.9:50690 - v1-completions-path 2026-11-03T22:25:21.885659Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:25:36.639469Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T22:25:44.478Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 96 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "b64143b8-5b39-49e2-8372-08f283efd61b" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60730 10.132.0.107:443 10.132.0.9:55042 - v1-completions-path [2026-11-03T22:25:45.456Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 96 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "fc75cf2d-90ea-4bf8-bac9-f01968c1b7b3" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60730 10.132.0.107:443 10.132.0.9:55062 - v1-completions-path [2026-11-03T22:25:47.256Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 101 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "ce36731d-b4d3-4d11-a08f-ef92915792d0" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60730 10.132.0.107:443 10.132.0.9:55068 - v1-completions-path 2026-11-03T22:26:00.862534Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:26:05.433536Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:26:17.112509Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:26:25.968693Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:26:33.536484Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:26:42.344964Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:27:08.066865Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:27:21.120993Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T22:27:43.390Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 56 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "f09bf37e-b4ad-43c8-b254-356f1149cf9f" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.107:443 10.132.0.9:42910 - v1-completions-path 2026-11-03T22:27:52.058621Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:28:13.572999Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:28:24.169020Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:31:43.949214Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-03T22:33:12.455423Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T22:33:12.455444Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T22:33:12.455447Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T22:33:12.455451Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T22:33:12.389Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 403 - - "-" 73 113 66 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "594c9d39-6f5a-4404-bbad-b9000138cfbf" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.107:443 10.132.0.9:46918 - v1-completions-path 2026-11-03T22:33:45.166870Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:33:49.354169Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T22:33:49.201Z] "DELETE /v1/api-keys/e2e-trlp-test-key HTTP/1.1" 404 - via_upstream - "-" 0 29 152 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "4ae4b575-5b3a-4642-b62a-ff7262fc803a" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.107:38412 10.132.0.107:443 10.132.0.9:49594 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T22:33:54.253688Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:34:06.056882Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:34:09.854667Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T22:34:09.854694Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T22:34:09.854698Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T22:34:09.854703Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-03T22:34:09.796Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 58 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "e679c612-8b46-4a68-ae46-2b6ec0e5af90" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.107:443 10.132.0.9:44178 - v1-completions-path 2026-11-03T22:34:23.385748Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:34:34.234887Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:34:54.840159Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:40:03.569315Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-11-03T22:42:01.316881Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:42:06.846608Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:43:53.472326Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:48:15.717779Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:48:20.792134Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:48:42.385984Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:48:53.117759Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:48:59.609701Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T22:49:09.328Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 98 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "aa0c833f-1f4d-43ed-87de-6f7a98f84400" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60730 10.132.0.107:443 10.132.0.9:39158 - v1-completions-path [2026-11-03T22:49:10.134Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 102 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "12a6d5aa-d296-4f16-a456-a7bd39468169" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60742 10.132.0.107:443 10.132.0.9:39172 - v1-completions-path 2026-11-03T22:49:26.059705Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:49:32.366058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:49:38.288054Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T22:49:56.871Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 98 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "410e36bf-f6f3-494f-b527-303215a1f73f" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60742 10.132.0.107:443 10.132.0.9:40476 - v1-completions-path 2026-11-03T22:49:58.526672Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T22:49:58.368Z] "DELETE /v1/api-keys/2ca72955-e8b7-4477-ad77-396f66529cf7 HTTP/1.1" 200 - via_upstream - "-" 0 365 157 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "89cdd978-fed6-4d96-8991-9d2bf32ce248" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.132.0.126:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.107:43694 10.132.0.107:443 10.132.0.9:40486 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T22:50:12.126093Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:50:22.959989Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:50:29.533276Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:50:35.465072Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:50:44.654069Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T22:50:51.508Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 106 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "222038bf-bbbf-4dd3-9ec3-1b24ed307695" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:60730 10.132.0.107:443 10.132.0.9:41504 - v1-completions-path 2026-11-03T22:51:07.880080Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:51:18.615596Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:51:23.514159Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:51:41.347840Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:51:52.136494Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:52:12.914236Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:52:34.907946Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:52:56.719414Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:53:16.586821Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:53:27.637704Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:53:38.378461Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:53:49.138151Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:53:59.821078Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T22:54:10.781469Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T22:54:12.969Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 287 616 - "44.212.242.249,10.132.0.9" "python-requests/2.33.0" "79c4d41e-69f8-4174-9aa2-94758b40c068" "maas.apps.88e1c44ca86628e62bf4.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.107:46536 10.132.0.107:443 10.132.0.9:53260 - v1-completions-path