2026-11-01T20:35:40.705257Z info FLAG: --concurrency="0" 2026-11-01T20:35:40.705274Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-01T20:35:40.705279Z info FLAG: --help="false" 2026-11-01T20:35:40.705281Z info FLAG: --log_as_json="false" 2026-11-01T20:35:40.705283Z info FLAG: --log_caller="" 2026-11-01T20:35:40.705285Z info FLAG: --log_output_level="default:info" 2026-11-01T20:35:40.705287Z info FLAG: --log_stacktrace_level="default:none" 2026-11-01T20:35:40.705292Z info FLAG: --log_target="[stdout]" 2026-11-01T20:35:40.705294Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-01T20:35:40.705296Z info FLAG: --outlierLogPath="" 2026-11-01T20:35:40.705298Z info FLAG: --profiling="true" 2026-11-01T20:35:40.705300Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-01T20:35:40.705302Z info FLAG: --proxyLogLevel="warning" 2026-11-01T20:35:40.705304Z info FLAG: --serviceCluster="istio-proxy" 2026-11-01T20:35:40.705306Z info FLAG: --stsPort="0" 2026-11-01T20:35:40.705308Z info FLAG: --templateFile="" 2026-11-01T20:35:40.705310Z info FLAG: --tokenManagerPlugin="" 2026-11-01T20:35:40.705313Z info FLAG: --vklog="0" 2026-11-01T20:35:40.705315Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-01T20:35:40.705459Z info Proxy role ips=[10.132.1.0] type=router id=maas-default-gateway-openshift-default-fd4b99879-qzbgz.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-01T20:35:40.705512Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-01T20:35:40.706621Z info cpu limit detected as 2, setting concurrency 2026-11-01T20:35:40.707194Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-01T20:35:40.707205Z info JWT policy is third-party-jwt 2026-11-01T20:35:40.707208Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-01T20:35:40.708702Z info Starting default Istio SDS Server 2026-11-01T20:35:40.708729Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-01T20:35:40.708756Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-01T20:35:40.709879Z info Opening status port 15020 2026-11-01T20:35:40.709971Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-01T20:35:40.710646Z info sds Starting SDS grpc server 2026-11-01T20:35:40.710665Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-01T20:35:40.712020Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-01T20:35:40.714632Z info Starting proxy agent 2026-11-01T20:35:40.714660Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-01T20:35:40.762456Z info cache generated new workload certificate resourceName=default latency=51.601087ms ttl=23h59m59.237546521s 2026-11-01T20:35:40.762491Z info cache Root cert has changed, start rotating root cert 2026-11-01T20:35:40.762533Z info cache returned workload trust anchor from cache ttl=23h59m59.237467911s 2026-11-01T20:35:40.779522Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-01T20:35:40.910936Z info ads ADS: new connection for node:1 2026-11-01T20:35:40.911003Z info cache returned workload certificate from cache ttl=23h59m59.088997768s 2026-11-01T20:35:40.911386Z info ads ADS: new connection for node:2 2026-11-01T20:35:40.911469Z info cache returned workload trust anchor from cache ttl=23h59m59.088532369s 2026-11-01T20:35:41.021803Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:35:49.171909Z info Readiness succeeded in 8.46930711s 2026-11-01T20:35:49.172393Z info Envoy proxy is ready 2026-11-01T20:37:46.380765Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:38:01.597262Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:38:04.094858Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:38:12.844583Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:38:17.906974Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:38:35.129110Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:38:54.929329Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:39:05.105511Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:39:15.475136Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:39:25.603223Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:39:35.877111Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:39:46.112543Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:39:56.337594Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:40:06.521692Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:40:16.915299Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:40:26.929231Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:41:52.495261Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T20:41:56.163Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 5518 5791 89 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "72a1d097-6b07-4fd3-aac5-c926bc377860" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.27:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:52534 10.132.1.0:443 10.132.0.21:59638 - v1-chat-completions-path 2026-11-01T20:42:08.177530Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:42:09.699911Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:42:13.376634Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-01T20:42:13.298Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 78 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "e0d8c487-cc40-4bcc-9592-707c1e790555" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.25:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:56080 10.132.1.0:443 10.132.0.21:55752 - v1-catch-all-path 2026-11-01T20:42:15.616711Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:42:30.910929Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:42:32.359711Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:42:46.523822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:42:57.505522Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:42:59.264571Z info ads ADS: new connection for node:3 2026-11-01T20:42:59.271241Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-01T20:42:59.271391Z info cache read certificate from file resource=file-root:system 2026-11-01T20:42:59.331443Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:43:00.341477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:44:12.055599Z info ads ADS: "" 3 terminated 2026-11-01T20:44:12.109299Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:44:12.586539Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:44:15.286703Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:44:17.092190Z info ads ADS: new connection for node:4 2026-11-01T20:44:17.092379Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T20:44:17.092409Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T20:44:18.655383Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:44:59.066603Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:45:02.030370Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:45:02.030413Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:45:02.030428Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:45:02.030434Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T20:45:01.887Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 142 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "5f6fde53-ffe9-4807-8c75-c4af9e985c29" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.0:46214 10.132.1.0:443 10.132.0.21:40152 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-01T20:45:02.897Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 328 86 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "11ac5988-18ed-4c67-87bb-4b0aa9921429" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.132.1.22:8000" outbound|54321||e2e-prio-pool-4cb35ada-inference-pool-ip-a775089e.llm.svc.cluster.local 10.132.1.0:53382 10.132.1.0:443 10.132.0.21:40178 - v1-model-routing 2026-11-01T20:45:17.611922Z info ads ADS: "" 4 terminated with stream closed 2026-11-01T20:45:17.660454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:45:19.296884Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:45:22.306032Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:45:26.600953Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:45:38.406259Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:45:55.239651Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:46:09.742697Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T20:46:09.742759Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T20:46:09.742810Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T20:46:09.742820Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T20:46:09.652Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 628 90 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "962c923a-f2a7-42d4-b604-dda41bdf41ab" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.0:40044 10.132.1.0:443 10.132.0.21:34288 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T20:46:10.827548Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:46:22.484932Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:46:25.600986Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:46:25.601018Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:46:25.601030Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-01T20:46:25.601036Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:46:26.274175Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T20:46:25.533Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 67 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "83e2b584-3d13-496e-8942-1b7a6510429b" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.0:43144 10.132.1.0:443 10.132.0.21:37274 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T20:46:36.968702Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:46:39.476419Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:46:54.699259Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:50:54.439299Z info ads ADS: new connection for node:5 2026-11-01T20:50:54.446513Z info cache read certificate from file resource=file-root:system 2026-11-01T20:50:54.932401Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:50:56.363260Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:52:17.396048Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-01T20:52:17.455062Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T20:52:17.455099Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T20:52:17.455116Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T20:52:17.455121Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-01T20:52:17.385Z] "DELETE /v1/api-keys/b24647ea-42a8-4bf2-b180-35828ef8a589 HTTP/1.1" 200 - via_upstream - "-" 0 359 70 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "90e121f3-35a9-4081-b82d-575c911d3d08" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.0:42354 10.132.1.0:443 10.132.0.21:33788 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T20:52:28.558569Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:52:28.699553Z info ads ADS: "" 5 terminated with stream closed 2026-11-01T20:52:29.108554Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:52:29.538087Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T20:52:29.538114Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T20:52:29.538118Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T20:52:29.538123Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-01T20:52:29.485Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 52 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "56945d4d-bb91-4154-8005-8df0fbcb6c8f" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.0:443 10.132.0.21:39478 - v1-completions-path 2026-11-01T20:52:32.388181Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:52:48.865039Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:52:52.080717Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:53:00.601880Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T20:53:06.043Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 73 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "d3e1b7f7-d07c-41cb-bf86-29c1ee67935c" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36664 10.132.1.0:443 10.132.0.21:59610 - v1-completions-path [2026-11-01T20:53:06.168Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 65 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "a6e8f69e-bdb2-4797-9106-0010664d4c91" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36674 10.132.1.0:443 10.132.0.21:59622 - v1-completions-path [2026-11-01T20:53:06.404Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 62 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "17deebae-01d9-4e2f-ac4e-731c1a10a173" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36664 10.132.1.0:443 10.132.0.21:59638 - v1-completions-path [2026-11-01T20:53:06.635Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 58 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "911a998f-1c4a-4a43-9ae1-c8b1f2719bb6" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36664 10.132.1.0:443 10.132.0.21:59642 - v1-completions-path 2026-11-01T20:53:21.913101Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:53:32.141589Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:53:33.659492Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T20:53:39.200Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 288 70 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "dc17f257-46cf-4cc7-b757-2236d54fb013" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36664 10.132.1.0:443 10.132.0.21:54040 - v1-completions-path [2026-11-01T20:53:40.711Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 59 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "2117f8e2-31af-42c3-b200-fc541d2984e8" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36664 10.132.1.0:443 10.132.0.21:44296 - v1-completions-path 2026-11-01T20:53:52.533955Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:53:54.039046Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:53:59.325040Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:54:10.234002Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:54:23.871564Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:54:25.322041Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:54:30.776196Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:54:49.011663Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:54:51.619355Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:55:05.510498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:55:06.993211Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:55:08.850106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T20:55:08.850148Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T20:55:08.850164Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T20:55:08.850169Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-01T20:55:08.768Z] "DELETE /v1/api-keys/4a126eb3-32c3-4fb5-86fa-044f7a9381d1 HTTP/1.1" 200 - via_upstream - "-" 0 375 81 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "74cf5e4e-810e-4416-9823-16862710ba24" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.0:36728 10.132.1.0:443 10.132.0.21:39338 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T20:55:12.344648Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:55:27.346124Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:55:37.585905Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:55:52.596743Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:55:55.052358Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:58:07.520838Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:58:09.973185Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:58:12.904282Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:58:27.943411Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:58:37.046460Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:58:41.078469Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:58:41.815582Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:58:41.815713Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:58:41.815773Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-01T20:58:41.815836Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-11-01T20:58:41.709Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 314 106 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "6cdd3b68-6618-4a66-b24d-faa9c9d3b11b" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.215:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.0:45834 10.132.1.0:443 10.132.0.21:40886 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-01T20:58:45.052Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 60 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "36539cb2-5054-48db-8a22-9e44a5feb6c6" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36664 10.132.1.0:443 10.132.0.21:40904 - v1-completions-path 2026-11-01T20:58:56.708363Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:59:07.056937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:59:09.966773Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:59:13.627623Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 2026-11-01T20:59:13.637323Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:59:13.637356Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:59:13.637371Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T20:59:13.637376Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T20:59:13.571Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 309 65 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "3ab5bbba-0d30-4533-a8b0-dc8ae07b2123" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.133.0.130:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.0:54566 10.132.1.0:443 10.132.0.21:49210 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-01T20:59:14.925Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 58 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "45a66dd1-0a92-46cc-8224-6827f4f20d0c" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36664 10.132.1.0:443 10.132.0.21:49212 - v1-completions-path 2026-11-01T20:59:26.518874Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:59:28.484344Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T20:59:31.784160Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T20:59:36.816Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "3bd5b055-cbf7-9655-a37c-85aa1e6550cf" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36674 10.132.1.0:443 10.132.0.21:52128 - v1-completions-path [2026-11-01T20:59:38.191Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 57 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "56b2c576-c8a0-49da-9f93-c3ae306fab75" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36674 10.132.1.0:443 10.132.0.21:52146 - v1-completions-path [2026-11-01T20:59:40.785Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 61 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "41d0143c-aeac-48d5-85e9-1948a1caba0f" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36674 10.132.1.0:443 10.132.0.21:50370 - v1-completions-path [2026-11-01T20:59:41.426Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 59 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "0263f929-3a39-4126-8608-03e0fc650bef" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.0:36674 10.132.1.0:443 10.132.0.21:50376 - v1-completions-path 2026-11-01T20:59:53.287355Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T21:00:03.579535Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T21:00:05.502275Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T21:00:08.833196Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T21:00:14.597970Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T21:00:33.160425Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T21:00:43.132673Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T21:00:43.132713Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T21:00:43.132730Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T21:00:43.132734Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T21:00:43.402728Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T21:00:43.063Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 331 69 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "a03d07da-77b5-4a8b-9ac1-4c0e288bedfa" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "10.133.0.130:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.0:56758 10.132.1.0:443 10.132.0.21:38780 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T21:00:46.100859Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T21:00:46.100887Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T21:00:46.100891Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T21:00:46.100896Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-01T21:00:46.071Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 29 - "44.212.242.249,10.132.0.21" "python-requests/2.33.0" "79e59bc2-1a18-4399-979d-d65237491535" "maas.apps.4ec91afccb26205cb961.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.0:443 10.132.0.21:38812 - v1-completions-path