2026-11-03T02:48:32.037805Z info FLAG: --concurrency="0" 2026-11-03T02:48:32.037825Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T02:48:32.037829Z info FLAG: --help="false" 2026-11-03T02:48:32.037832Z info FLAG: --log_as_json="false" 2026-11-03T02:48:32.037834Z info FLAG: --log_caller="" 2026-11-03T02:48:32.037836Z info FLAG: --log_output_level="default:info" 2026-11-03T02:48:32.037838Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T02:48:32.037845Z info FLAG: --log_target="[stdout]" 2026-11-03T02:48:32.037848Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T02:48:32.037850Z info FLAG: --outlierLogPath="" 2026-11-03T02:48:32.037851Z info FLAG: --profiling="true" 2026-11-03T02:48:32.037853Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T02:48:32.037855Z info FLAG: --proxyLogLevel="warning" 2026-11-03T02:48:32.037857Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T02:48:32.037859Z info FLAG: --stsPort="0" 2026-11-03T02:48:32.037861Z info FLAG: --templateFile="" 2026-11-03T02:48:32.037863Z info FLAG: --tokenManagerPlugin="" 2026-11-03T02:48:32.037866Z info FLAG: --vklog="0" 2026-11-03T02:48:32.037868Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T02:48:32.038043Z info Proxy role ips=[10.133.1.123] type=router id=maas-default-gateway-openshift-default-59f957fdd7-4jmtt.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T02:48:32.038093Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T02:48:32.039299Z info cpu limit detected as 2, setting concurrency 2026-11-03T02:48:32.039566Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T02:48:32.039578Z info JWT policy is third-party-jwt 2026-11-03T02:48:32.039581Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T02:48:32.041387Z info Starting default Istio SDS Server 2026-11-03T02:48:32.041409Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T02:48:32.041426Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T02:48:32.041676Z info Opening status port 15020 2026-11-03T02:48:32.042637Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T02:48:32.043528Z info sds Starting SDS grpc server 2026-11-03T02:48:32.043546Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T02:48:32.044043Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T02:48:32.044926Z info Starting proxy agent 2026-11-03T02:48:32.044976Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T02:48:32.111830Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T02:48:32.153030Z info cache generated new workload certificate resourceName=default latency=109.347733ms ttl=23h59m59.846975356s 2026-11-03T02:48:32.153081Z info cache Root cert has changed, start rotating root cert 2026-11-03T02:48:32.153122Z info cache returned workload trust anchor from cache ttl=23h59m59.846878113s 2026-11-03T02:48:32.265627Z info ads ADS: new connection for node:1 2026-11-03T02:48:32.265713Z info cache returned workload certificate from cache ttl=23h59m59.734289803s 2026-11-03T02:48:32.265872Z info ads ADS: new connection for node:2 2026-11-03T02:48:32.265940Z info cache returned workload trust anchor from cache ttl=23h59m59.734061381s 2026-11-03T02:48:32.354441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:48:40.331761Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:48:41.258183Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-03T02:48:41.548532Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-03T02:48:42.730360Z info Readiness succeeded in 10.695153717s 2026-11-03T02:48:42.730678Z info Envoy proxy is ready 2026-11-03T02:49:10.612536Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T02:49:12.857Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 147 412 104 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "c00f9205-08a1-404e-9276-141f5ac1b75c" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:45172 10.133.1.123:443 10.132.0.19:48698 - v1-chat-completions-path [2026-11-03T02:49:13.530Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 108118 108394 324 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "fb6e3de3-ae4e-4919-922a-9ad52bd825a2" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:45172 10.133.1.123:443 10.132.0.19:48704 - v1-chat-completions-path 2026-11-03T02:49:27.979476Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T02:49:27.979502Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T02:49:27.979506Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T02:49:27.979511Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-03T02:49:27.835Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 144 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "82ca5344-6b24-4493-8245-4844d425a10e" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.123:443 10.132.0.19:55766 - v1-catch-all-path 2026-11-03T02:49:29.212545Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:49:33.839879Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:49:43.708407Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:49:48.230451Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T02:49:48.130Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 100 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "e4bb91e0-5cad-445a-89a0-8407bbd94943" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:54310 10.133.1.123:443 10.132.0.19:41528 - v1-catch-all-path 2026-11-03T02:49:49.341981Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-03T02:49:49.419025Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T02:49:49.334Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 84 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "7128d827-483a-44e4-a290-c0ec0a1d0fd0" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:54310 10.133.1.123:443 10.132.0.19:41536 - v1-catch-all-path 2026-11-03T02:49:57.623680Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:50:06.754597Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:50:09.935638Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T02:50:09.935674Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T02:50:09.935697Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T02:50:09.935702Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T02:50:09.820Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 115 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "66d475c5-8a00-44a7-87b3-9fd8e076e029" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.123:53858 10.133.1.123:443 10.132.0.19:51368 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T02:50:16.530028Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:50:35.843854Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:50:39.044067Z info ads ADS: new connection for node:3 2026-11-03T02:50:39.044337Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T02:50:39.044386Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T02:50:43.009712Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:51:22.224122Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:51:46.502466Z info ads ADS: "" 3 terminated with stream closed 2026-11-03T02:51:46.790830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:51:51.522175Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:51:51.522214Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:51:51.522228Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:51:51.522233Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T02:51:51.410Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 90 332 111 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "867aaac4-0655-4a88-8fe7-e2808b914c3f" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.123:35990 10.133.1.123:443 10.132.0.19:36578 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T02:51:53.493306Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:52:08.660794Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:54:24.761257Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:54:24.761296Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:54:24.761315Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T02:54:24.761319Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T02:54:24.638Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 33 325 123 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "a7d9d6ea-91d2-467e-a3aa-0ea663d8e1b4" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.123:53202 10.133.1.123:443 10.132.0.19:40060 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T02:54:25.237908Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:54:25.237942Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:54:25.237955Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T02:54:25.237960Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T02:54:25.171Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 66 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "108ab65d-ac00-4b0c-9c2c-b3399767e403" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.123:53202 10.133.1.123:443 10.132.0.19:40074 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T02:54:43.433512Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:55:10.390173Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:55:32.810509Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:55:48.148790Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:55:56.824629Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:56:10.406390Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:56:15.266463Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:56:33.910141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T02:56:37.000Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 278 76 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "19ffa38d-167c-463d-a8f3-12b62b21b1f3" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:44884 10.133.1.123:443 10.132.0.19:57552 - v1-completions-path [2026-11-03T02:57:20.964Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 70 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "8bbeaed9-4aa5-47e1-a8e9-adecaf118ff3" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:44884 10.133.1.123:443 10.132.0.19:45446 - v1-completions-path 2026-11-03T02:57:22.784389Z info ads ADS: new connection for node:4 2026-11-03T02:57:22.788217Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T02:57:22.788264Z info cache read certificate from file resource=file-root:system 2026-11-03T02:57:26.552760Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:57:27.899507Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:57:56.881857Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:57:56.881889Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:57:56.881899Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T02:57:56.881903Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:57:56.882063Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T02:57:56.681Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1720 200 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "6dcd37a6-11f1-4af7-ab70-bd7968a24a3d" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.123:58958 10.133.1.123:443 10.132.0.19:45054 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T02:58:06.374296Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:58:06.374343Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:58:06.374382Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T02:58:06.374389Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T02:58:06.374549Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T02:58:06.255Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1723 119 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "01db4c9b-0a63-47a9-b8b3-fcdcaaaa5ec7" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.123:58958 10.133.1.123:443 10.132.0.19:38376 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T02:58:32.487268Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T02:58:32.487304Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T02:58:32.487316Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T02:58:32.487322Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T02:58:32.487549Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T02:58:32.285Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1723 202 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "caa846b0-3ac1-4bc7-a299-4e1ef0157efb" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.123:33262 10.133.1.123:443 10.132.0.19:45292 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T02:59:02.580235Z info ads ADS: "" 4 terminated with stream closed 2026-11-03T02:59:02.638119Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:59:05.292529Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:59:07.408510Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T02:59:07.408537Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T02:59:07.408541Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T02:59:07.408546Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T02:59:07.353Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 54 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "ac4495be-3263-4515-8c48-2a2bea6272e6" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.123:443 10.132.0.19:47948 - v1-completions-path 2026-11-03T02:59:15.954844Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:59:34.541901Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T02:59:39.317010Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T02:59:44.752Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 59 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "5a72934a-f782-4b22-af5f-351453d8ee73" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.123:443 10.132.0.19:50140 - v1-completions-path 2026-11-03T02:59:58.678919Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:00:16.173241Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T03:00:24.991Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 71 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "fc1e0985-b161-41d1-b27c-799afecd311b" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:42978 10.133.1.123:443 10.132.0.19:35770 - v1-completions-path 2026-11-03T03:00:46.795552Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:00:57.501642Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:01:03.899930Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:01:32.921442Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:01:37.443060Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:01:49.244875Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:01:58.035606Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:02:07.589102Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:02:16.475572Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:02:24.456064Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:02:43.273620Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:03:14.952708Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:03:23.915373Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:03:23.915410Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T03:03:23.915425Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T03:03:23.915430Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T03:03:23.804Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 65 308 111 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "7341a746-f3da-47bc-bd7c-8eeda84daee2" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.123:57682 10.133.1.123:443 10.132.0.19:34960 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T03:03:38.269494Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:03:48.935741Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:08:15.966815Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:08:33.394052Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:08:37.913538Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T03:08:41.250Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 68 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "ae684939-7079-9f2b-ba8c-b23061e3a889" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:44884 10.133.1.123:443 10.132.0.19:50152 - v1-completions-path 2026-11-03T03:08:51.012512Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:09:08.603540Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:09:16.469552Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:09:36.974316Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:09:47.749026Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:09:54.013818Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:10:05.513096Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-03T03:10:05.448Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 71 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "b6a58213-03fa-9174-abf0-6ca5d5c84184" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:48472 10.133.1.123:443 10.132.0.19:43522 - v1-completions-path [2026-11-03T03:10:07.075Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 62 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "c2341d9a-26d5-416f-9e0a-de324a04907b" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:48472 10.133.1.123:443 10.132.0.19:43542 - v1-completions-path 2026-11-03T03:10:22.038760Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:10:28.336411Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:10:34.341318Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T03:10:48.535Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 65 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "1ec7c7ce-036b-4c5c-ad01-3089ee10a793" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.123:48472 10.133.1.123:443 10.132.0.19:60784 - v1-completions-path 2026-11-03T03:10:51.416363Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T03:10:51.416405Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T03:10:51.416422Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T03:10:51.416427Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T03:10:51.297Z] "DELETE /v1/api-keys/5b4fd0d9-0545-41b7-be06-feb0eba61a00 HTTP/1.1" 200 - via_upstream - "-" 0 363 118 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "d8f848f2-5386-46e4-a231-6064588d85d4" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "10.133.1.160:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.123:45742 10.133.1.123:443 10.132.0.19:34624 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T03:11:05.326066Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:11:15.940874Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:11:22.513693Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:11:28.554148Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:11:39.273539Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T03:11:48.944Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 87 - "34.228.250.194,10.132.0.19" "python-requests/2.34.2" "02275a13-8e18-4847-83ad-6c8be4c4f2cd" "maas.apps.a6ff95811a9faa36af00.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.123:443 10.132.0.19:44562 - v1-completions-path 2026-11-03T03:12:02.922822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:12:13.741464Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:12:18.658900Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:12:37.032447Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:12:44.133752Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:13:04.784488Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:13:26.723313Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:13:46.765834Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:13:57.719835Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:14:08.373325Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:14:19.195059Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:14:30.131007Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:14:40.921102Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:14:51.650995Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T03:15:02.371892Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8