2026-11-03T04:06:48.799870Z info FLAG: --concurrency="0" 2026-11-03T04:06:48.799890Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T04:06:48.799894Z info FLAG: --help="false" 2026-11-03T04:06:48.799896Z info FLAG: --log_as_json="false" 2026-11-03T04:06:48.799898Z info FLAG: --log_caller="" 2026-11-03T04:06:48.799900Z info FLAG: --log_output_level="default:info" 2026-11-03T04:06:48.799902Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T04:06:48.799908Z info FLAG: --log_target="[stdout]" 2026-11-03T04:06:48.799910Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T04:06:48.799912Z info FLAG: --outlierLogPath="" 2026-11-03T04:06:48.799914Z info FLAG: --profiling="true" 2026-11-03T04:06:48.799916Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T04:06:48.799918Z info FLAG: --proxyLogLevel="warning" 2026-11-03T04:06:48.799920Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T04:06:48.799922Z info FLAG: --stsPort="0" 2026-11-03T04:06:48.799923Z info FLAG: --templateFile="" 2026-11-03T04:06:48.799925Z info FLAG: --tokenManagerPlugin="" 2026-11-03T04:06:48.799928Z info FLAG: --vklog="0" 2026-11-03T04:06:48.799930Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T04:06:48.800065Z info Proxy role ips=[10.132.0.132] type=router id=maas-default-gateway-openshift-default-76f4876667-lbdzp.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T04:06:48.800121Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T04:06:48.801528Z info cpu limit detected as 2, setting concurrency 2026-11-03T04:06:48.801984Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T04:06:48.802003Z info JWT policy is third-party-jwt 2026-11-03T04:06:48.802006Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T04:06:48.804031Z info Starting default Istio SDS Server 2026-11-03T04:06:48.804052Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T04:06:48.804020Z info Opening status port 15020 2026-11-03T04:06:48.804082Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T04:06:48.804757Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T04:06:48.804898Z info sds Starting SDS grpc server 2026-11-03T04:06:48.804913Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T04:06:48.806282Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T04:06:48.807208Z info Starting proxy agent 2026-11-03T04:06:48.807248Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T04:06:48.869786Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T04:06:48.891994Z info cache generated new workload certificate resourceName=default latency=86.984002ms ttl=23h59m59.108008692s 2026-11-03T04:06:48.892030Z info cache Root cert has changed, start rotating root cert 2026-11-03T04:06:48.892058Z info cache returned workload trust anchor from cache ttl=23h59m59.107942091s 2026-11-03T04:06:49.080969Z info ads ADS: new connection for node:1 2026-11-03T04:06:49.088011Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T04:06:49.088063Z info cache read certificate from file resource=file-root:system 2026-11-03T04:06:49.113650Z info ads ADS: new connection for node:2 2026-11-03T04:06:49.113725Z info cache returned workload certificate from cache ttl=23h59m58.886276792s 2026-11-03T04:06:49.114680Z info ads ADS: new connection for node:3 2026-11-03T04:06:49.114753Z info cache returned workload trust anchor from cache ttl=23h59m58.885248373s 2026-11-03T04:06:49.208774Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:06:57.427561Z info Readiness succeeded in 8.629944876s 2026-11-03T04:06:57.427930Z info Envoy proxy is ready 2026-11-03T04:07:35.572606Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:07:44.145318Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T04:07:44.145349Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T04:07:44.145357Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T04:07:44.145361Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T04:07:44.145546Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T04:07:43.900Z] "POST /anything/e2e-dft-700af7-8f89de HTTP/1.1" 200 - via_upstream - "-" 109 1771 245 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "65278b66-8bf8-412c-832c-68727209b041" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.132:42132 10.132.0.132:443 10.132.0.10:37028 - llm.external-model-e2e-dft-700af7-c323723dac88.0 2026-11-03T04:07:58.238617Z info ads ADS: "" 1 terminated with stream closed 2026-11-03T04:07:59.235154Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:08:01.472196Z info ads ADS: new connection for node:4 2026-11-03T04:08:01.477674Z info cache read certificate from file resource=file-root:system 2026-11-03T04:08:05.913790Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:08:07.298771Z info ads ADS: "" 4 terminated 2026-11-03T04:08:09.686874Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:08:14.017679Z info ads ADS: new connection for node:5 2026-11-03T04:08:14.027263Z info cache read certificate from file resource=file-root:system 2026-11-03T04:08:19.016053Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:08:20.272042Z info ads ADS: "" 5 terminated 2026-11-03T04:08:22.933635Z info ads ADS: new connection for node:6 2026-11-03T04:08:22.940864Z info cache read certificate from file resource=file-root:system 2026-11-03T04:08:23.948468Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:08:29.393313Z info ads ADS: "" 6 terminated 2026-11-03T04:08:29.995382Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:13:11.701793Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:14:14.969955Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:16:55.263987Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:18:36.291376Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:27:52.054741Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:28:19.179318Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:28:55.431781Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:28:59.367Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 5518 5791 85 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "d8267cfb-19eb-4022-88e3-199ba84dc45a" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:47976 10.132.0.132:443 10.132.0.10:49660 - v1-chat-completions-path [2026-11-03T04:29:00.367Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 144 409 68 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "f36a2187-e5ad-4477-a08c-e916c64697ba" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:47980 10.132.0.132:443 10.132.0.10:49676 - v1-chat-completions-path 2026-11-03T04:29:13.015946Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T04:29:13.016087Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T04:29:13.016129Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T04:29:13.016157Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T04:29:12.857Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 158 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "f6f28731-9eb1-4598-a48e-0fbbc315bd8b" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.132:443 10.132.0.10:55450 - v1-catch-all-path 2026-11-03T04:29:14.649119Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:29:19.958398Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:29:20.315320Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:29:20.315360Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:29:20.315378Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T04:29:20.315384Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T04:29:20.137Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 324 177 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "4d4a61cd-4770-40cd-924a-81f39aeb95a9" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.132:34556 10.132.0.132:443 10.132.0.10:52744 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T04:29:29.758015Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:29:32.934686Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-03T04:29:32.936544Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 2026-11-03T04:29:33.293247Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T04:29:32.454Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 482 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "e16e756e-9f3b-4642-80ac-52a66c3fadc9" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:50030 10.132.0.132:443 10.132.0.10:49002 - v1-catch-all-path [2026-11-03T04:29:33.232Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 61 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "17b43a85-6351-4ad4-ada9-e6b5edf3dfdc" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:50030 10.132.0.132:443 10.132.0.10:39650 - v1-catch-all-path 2026-11-03T04:29:35.167928Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T04:29:35.107Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 60 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "b012c3e4-8036-45fc-a9c5-5176fa9ddca3" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:50030 10.132.0.132:443 10.132.0.10:39652 - v1-catch-all-path 2026-11-03T04:29:43.965601Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:29:53.448056Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:30:10.735402Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:30:22.569276Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:30:26.597226Z info ads ADS: new connection for node:7 2026-11-03T04:30:26.597379Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T04:30:26.597403Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T04:30:30.162586Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:31:13.964521Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:31:41.099387Z info ads ADS: "" 7 terminated 2026-11-03T04:31:41.448771Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:31:46.473025Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:32:04.326544Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:35:01.042624Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:35:27.122017Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:35:51.780271Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:36:07.870266Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:36:16.998080Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:36:19.139270Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:36:19.139307Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:36:19.139321Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:36:19.139326Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T04:36:18.640Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 84 326 498 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "6b681a03-dac6-4a9e-84e5-62d8f7dbe90f" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.132:50942 10.132.0.132:443 10.132.0.10:53564 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T04:36:23.905764Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:36:33.831982Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:36:38.555371Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:36:42.339Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 80 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "7e530565-75a7-935f-a034-ae9d9afadef7" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:54314 10.132.0.132:443 10.132.0.10:44828 - v1-completions-path [2026-11-03T04:36:43.175Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 60 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "7ca7c5ab-46f7-4845-9f66-1c01f5b2213c" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:52170 10.132.0.132:443 10.132.0.10:37786 - v1-completions-path 2026-11-03T04:36:58.496456Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:37:45.825Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 85 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "1217f68c-0b3d-455f-b216-436a195d1611" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.41:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:57256 10.132.0.132:443 10.132.0.10:37448 - v1-completions-path 2026-11-03T04:37:47.623754Z info ads ADS: new connection for node:8 2026-11-03T04:37:47.630336Z info cache read certificate from file resource=file-root:system 2026-11-03T04:37:51.511826Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:37:52.985691Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:38:03.167591Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T04:38:03.171430Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"543","StreamId":"1866131484252744288"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-03T04:38:03.107Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 64 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "b371aa8c-51a7-4072-ae0f-f479361988cf" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.0.132:443 10.132.0.10:58626 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T04:38:12.687849Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:38:12.687880Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:38:12.687894Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:38:12.687901Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T04:38:12.688076Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T04:38:12.562Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1713 126 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "c6aba4d4-0e5d-4677-ac63-88300a60f5c1" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.132:53690 10.132.0.132:443 10.132.0.10:41516 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T04:38:44.728996Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-03T04:39:13.214182Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:39:13.214220Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:39:13.214235Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T04:39:13.214241Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:39:13.214425Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T04:39:13.100Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 113 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "4936b312-969a-4c88-9d11-4599feeae78e" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.132:53690 10.132.0.132:443 10.132.0.10:35926 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T04:39:28.621268Z info ads ADS: "" 8 terminated 2026-11-03T04:39:28.665010Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:39:31.331505Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:39:42.595604Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:39:43.343896Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T04:39:43.343950Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T04:39:43.343978Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T04:39:43.343986Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T04:39:43.144Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 87 329 199 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "6df395f1-1459-43ff-a777-b94e31a541b4" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.132:52472 10.132.0.132:443 10.132.0.10:43192 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T04:39:45.127Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 603 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "b9a11df5-a160-4cf3-96b1-15d2025fcae2" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:54314 10.132.0.132:443 10.132.0.10:58406 - v1-completions-path 2026-11-03T04:40:01.579721Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:40:06.465273Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:40:10.517Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 64 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "4b97b6c7-ee4f-460d-a34c-d3511e16f6aa" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:54314 10.132.0.132:443 10.132.0.10:39068 - v1-completions-path [2026-11-03T04:40:13.207Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 58 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "8a016cae-8e0c-4113-9072-b4f93c1b5328" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:54314 10.132.0.132:443 10.132.0.10:57018 - v1-completions-path 2026-11-03T04:40:27.739361Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:40:44.645616Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:40:55.362Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 61 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "24ade2d9-80bb-4f4a-b977-e8188bdfa3d1" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:54314 10.132.0.132:443 10.132.0.10:50226 - v1-completions-path [2026-11-03T04:40:55.893Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 62 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "7f5efa70-751e-4304-b12d-557de638c416" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:52170 10.132.0.132:443 10.132.0.10:50230 - v1-completions-path 2026-11-03T04:41:16.051332Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:41:27.054407Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:41:31.829555Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:41:44.005550Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:41:44.005586Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:41:44.005601Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T04:41:44.005606Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T04:41:43.890Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 317 115 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "71376f5c-7cb9-44a2-b625-5649101ac30b" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.132:50498 10.132.0.132:443 10.132.0.10:37218 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T04:41:47.901Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 58 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "ecea985e-3cc2-47d0-858c-71afb5b0dd52" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:52170 10.132.0.132:443 10.132.0.10:37244 - v1-completions-path 2026-11-03T04:42:01.975120Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:42:06.838815Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:42:21.068175Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:42:30.437786Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:42:38.761984Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:42:48.329109Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:42:58.426315Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:43:15.735384Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:43:19.063Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 66 18 60 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "4fd08ad2-8e65-420d-8bac-07e87e49cc10" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.132:443 10.132.0.10:60006 - v1-completions-path 2026-11-03T04:43:22.156854Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:43:21.336Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 66 18 82 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "a0fdd855-0712-4e8b-b4ac-a1f06bedb4e3" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.132:443 10.132.0.10:60022 - v1-completions-path [2026-11-03T04:43:24.821Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 312 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "730380a9-cd69-4c04-8182-af541082d14e" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.41:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:37996 10.132.0.132:443 10.132.0.10:51396 - v1-completions-path [2026-11-03T04:43:42.973Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 64 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "2397e238-5f1b-42b4-9757-69e405ba2a57" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.41:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:57256 10.132.0.132:443 10.132.0.10:46184 - v1-completions-path [2026-11-03T04:43:43.239Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 32 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "de199e51-1d75-4000-a6c4-059373fae394" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.132:443 10.132.0.10:37332 - v1-completions-path 2026-11-03T04:43:43.888912Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:43:43.888951Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:43:43.888967Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:43:43.888973Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T04:43:43.777Z] "DELETE /v1/api-keys/67d07191-abbe-4eae-adec-389f2bbe9f86 HTTP/1.1" 200 - via_upstream - "-" 0 375 111 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "26bfaee6-3d06-4bdd-bd1d-70e671487fee" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.132:43672 10.132.0.132:443 10.132.0.10:37338 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T04:43:53.859023Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:44:03.230Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 299 66 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "1857b360-697c-442b-af2e-ce65d297d163" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:52170 10.132.0.132:443 10.132.0.10:40078 - v1-completions-path 2026-11-03T04:44:17.141513Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:44:28.466900Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:44:47.303763Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:44:55.551438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:48:28.395190Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:48:28.395227Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:48:28.395243Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:48:28.395248Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T04:48:28.279Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 313 116 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "77a5dbcf-b091-4296-bfe8-4891261da47e" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.132:40916 10.132.0.132:443 10.132.0.10:53430 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T04:48:49.661090Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:48:52.070Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 73 297 307 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "f5766f40-53f8-4b3d-b5fa-76b5e428c5a5" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.39:8000" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:55322 10.132.0.132:443 10.132.0.10:57768 - v1-completions-path 2026-11-03T04:49:07.049804Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:49:11.748036Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:49:25.508906Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:49:42.067710Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:49:47.191804Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:50:07.472980Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:50:18.222895Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:50:24.718989Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:50:36.796551Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T04:50:36.796595Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T04:50:36.796617Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T04:50:36.796624Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T04:50:36.677Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 310 119 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "ea7e9ca3-e488-4797-a500-b47a6cb8dae7" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.243:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.132:42482 10.132.0.132:443 10.132.0.10:59366 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T04:50:53.912031Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:51:00.481467Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:51:06.920161Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:51:15.753Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "af5fefb6-2606-94df-966b-e066c307c218" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:52170 10.132.0.132:443 10.132.0.10:59018 - v1-completions-path [2026-11-03T04:51:20.673Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 60 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "25c24f8a-7f1f-42bb-9db9-8b20bdb1ebf1" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:54314 10.132.0.132:443 10.132.0.10:59036 - v1-completions-path [2026-11-03T04:51:24.710Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 65 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "1e881c68-a591-4ff6-abb9-493299b494f8" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.132:443 10.132.0.10:46702 - v1-completions-path 2026-11-03T04:51:39.020383Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:51:49.706809Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:51:56.667236Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:52:03.120024Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:52:11.292Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 64 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "aad65548-81fa-4aae-b5cd-e97904dcf7ad" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:54314 10.132.0.132:443 10.132.0.10:38122 - v1-completions-path 2026-11-03T04:52:14.946766Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:52:22.203Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 63 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "92412aa5-9b64-4155-af1e-bb5edacfdbfe" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.132:52170 10.132.0.132:443 10.132.0.10:60168 - v1-completions-path 2026-11-03T04:52:24.758209Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:52:24.758247Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:52:24.758262Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T04:52:24.758267Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T04:52:24.646Z] "DELETE /v1/api-keys/2a4a7e36-87fd-4e6b-a410-6740c7a977b0 HTTP/1.1" 200 - via_upstream - "-" 0 367 111 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "7532c1b9-6b07-45e7-972c-150f420a3b88" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.243:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.132:60218 10.132.0.132:443 10.132.0.10:56280 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T04:52:38.398191Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:52:49.244992Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:52:52.609920Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:53:12.376342Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:53:18.788630Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:53:42.622751Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:54:02.804502Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:54:13.483430Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:54:26.096574Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:54:45.991605Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:54:50.277508Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-11-03T04:54:56.692309Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:55:07.463115Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:55:18.345543Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:55:28.954384Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-03T04:55:39.272973Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T04:55:39.273009Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T04:55:39.273023Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T04:55:39.273029Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T04:55:39.737319Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-03T04:55:39.154Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 331 118 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "2dd012de-cec9-47bf-920e-94c37d541358" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "10.134.0.243:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.132:54154 10.132.0.132:443 10.132.0.10:45830 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T04:55:42.164375Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T04:55:42.164403Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T04:55:42.164406Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T04:55:42.164410Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T04:55:42.143Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 20 - "34.228.250.194,10.132.0.10" "python-requests/2.34.2" "a72bcb21-53fe-4c71-abcf-1cdc715c90da" "maas.apps.bbf32bbb4c5c76878149.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.132:443 10.132.0.10:45856 - v1-completions-path