2026-10-30T03:33:30.698946Z info FLAG: --concurrency="0" 2026-10-30T03:33:30.698963Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T03:33:30.698967Z info FLAG: --help="false" 2026-10-30T03:33:30.698969Z info FLAG: --log_as_json="false" 2026-10-30T03:33:30.698971Z info FLAG: --log_caller="" 2026-10-30T03:33:30.698973Z info FLAG: --log_output_level="default:info" 2026-10-30T03:33:30.698976Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T03:33:30.698980Z info FLAG: --log_target="[stdout]" 2026-10-30T03:33:30.698982Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T03:33:30.698984Z info FLAG: --outlierLogPath="" 2026-10-30T03:33:30.698986Z info FLAG: --profiling="true" 2026-10-30T03:33:30.698988Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T03:33:30.698990Z info FLAG: --proxyLogLevel="warning" 2026-10-30T03:33:30.698992Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T03:33:30.698994Z info FLAG: --stsPort="0" 2026-10-30T03:33:30.698996Z info FLAG: --templateFile="" 2026-10-30T03:33:30.698998Z info FLAG: --tokenManagerPlugin="" 2026-10-30T03:33:30.699001Z info FLAG: --vklog="0" 2026-10-30T03:33:30.699003Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T03:33:30.699206Z info Proxy role ips=[10.133.0.40] type=router id=maas-default-gateway-openshift-default-64d4cc998c-p77bg.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T03:33:30.699260Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T03:33:30.700473Z info cpu limit detected as 2, setting concurrency 2026-10-30T03:33:30.700741Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T03:33:30.700753Z info JWT policy is third-party-jwt 2026-10-30T03:33:30.700756Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T03:33:30.702691Z info Starting default Istio SDS Server 2026-10-30T03:33:30.702863Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T03:33:30.703224Z info Opening status port 15020 2026-10-30T03:33:30.703237Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T03:33:30.703988Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T03:33:30.705100Z info sds Starting SDS grpc server 2026-10-30T03:33:30.705148Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T03:33:30.705381Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T03:33:30.706199Z info Starting proxy agent 2026-10-30T03:33:30.706230Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T03:33:30.770070Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T03:33:30.798029Z info cache generated new workload certificate resourceName=default latency=92.815564ms ttl=23h59m59.201973225s 2026-10-30T03:33:30.798061Z info cache Root cert has changed, start rotating root cert 2026-10-30T03:33:30.798098Z info cache returned workload trust anchor from cache ttl=23h59m59.201902278s 2026-10-30T03:33:30.889676Z info ads ADS: new connection for node:1 2026-10-30T03:33:30.889730Z info cache returned workload certificate from cache ttl=23h59m59.110271148s 2026-10-30T03:33:30.890050Z info ads ADS: new connection for node:2 2026-10-30T03:33:30.890131Z info cache returned workload trust anchor from cache ttl=23h59m59.109877166s 2026-10-30T03:33:30.961897Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:33:31.012901Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:33:31.012921Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:341 createWasm: failed to load (in progress) from http://kuadrant-operator-wasm.kuadrant-system.svc.cluster.local:8082/plugin.wasm thread=8 2026-10-30T03:33:31.012926Z critical envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:615 Plugin kuadrant-wasm-shim failed to load thread=8 2026-10-30T03:33:31.012933Z critical envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:465 Plugin configured to fail closed failed to load thread=8 2026-10-30T03:33:31.012937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:350 createWasm: failed to load (cached) from http://kuadrant-operator-wasm.kuadrant-system.svc.cluster.local:8082/plugin.wasm thread=8 2026-10-30T03:33:31.012939Z critical envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:615 Plugin kuadrant-wasm-shim failed to load thread=8 2026-10-30T03:33:31.012942Z critical envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:465 Plugin configured to fail closed failed to load thread=8 2026-10-30T03:33:38.609203Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:33:40.579030Z info Readiness succeeded in 9.882390269s 2026-10-30T03:33:40.579418Z info Envoy proxy is ready 2026-10-30T03:33:40.580797Z info Readiness succeeded in 9.884165759s 2026-10-30T03:33:55.454309Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T03:34:05.509Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 531 87 - "52.71.36.72,10.132.0.20" "curl/7.76.1" "25eedf63-17a1-470a-a645-bf57274e36fd" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.56:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:39894 10.133.0.40:443 10.132.0.20:58282 - v1-model-routing [2026-10-30T03:34:05.626Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 339 70 - "52.71.36.72,10.132.0.20" "curl/7.76.1" "aed28d87-6afa-4f59-919d-fb4852d35e2c" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.56:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:39894 10.133.0.40:443 10.132.0.20:58298 - v1-model-routing [2026-10-30T03:34:06.317Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 63 - "52.71.36.72,10.132.0.20" "curl/7.76.1" "1333e924-62ae-45c5-83e1-4c7269f49339" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.40:443 10.132.0.20:58302 - v1-model-routing 2026-10-30T03:40:31.226402Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:40:51.686576Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:42:42.178370Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:43:44.560498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:45:09.293740Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T03:45:09.293783Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T03:45:09.293799Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T03:45:09.293805Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T03:45:09.201Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 92 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "5974f661-c181-469d-8901-a37449ea0d0d" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.40:56874 10.133.0.40:443 10.132.0.20:51414 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T03:45:11.356Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 71 - "52.71.36.72,10.132.0.20" "-" "aaeae48f-27a2-4124-964b-fc9c0f10c853" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.56:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:46738 10.133.0.40:443 10.132.0.20:51428 - v1-completions-path 2026-10-30T03:49:40.418991Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T03:49:40.419034Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T03:49:40.419054Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T03:49:40.419060Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T03:49:40.294Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 69 311 124 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "d051e5b4-d47c-46df-8ebc-bffa5ff40d7f" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.40:38462 10.133.0.40:443 10.132.0.20:56406 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T03:52:20.439851Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T03:52:20.439885Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T03:52:20.439897Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T03:52:20.439902Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T03:52:20.425Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 14 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "f244fb5a-3752-421c-b515-926bb6e62695" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.40:49586 10.133.0.40:443 10.132.0.20:53022 - odh-ai-gateway-infra.maas-api-route.4 [2026-10-30T03:55:07.853Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 113 323 66 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "47c41f8d-3ea4-402d-ae48-7e6e7902d4c9" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.56:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:46738 10.133.0.40:443 10.132.0.20:59592 - v1-chat-completions-path 2026-10-30T04:02:38.067625Z info ads ADS: new connection for node:3 2026-10-30T04:02:38.074688Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T04:02:38.074738Z info cache read certificate from file resource=file-root:system 2026-10-30T04:06:09.098093Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T04:07:26.188726Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:07:45.479999Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:07:48.468658Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-10-30T04:07:48.521089Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:07:48.521125Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:07:48.521142Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:07:48.521148Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T04:07:48.521343Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T04:07:48.306Z] "POST /anything/e2e-dft-d9f9cd-837b1e HTTP/1.1" 200 - via_upstream - "-" 109 1750 215 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e4bb0d47-1dd0-475c-8bcb-184686c4de60" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.40:48468 10.133.0.40:443 10.132.0.20:46702 - llm.external-model-e2e-dft-d9f9cd-b33c99c7ac48.0 2026-10-30T04:08:00.451266Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:08:11.300421Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:08:16.150177Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:08:18.065475Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:08:21.481182Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:08:21.607549Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:08:56.755654Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:12:13.368561Z info ads ADS: new connection for node:4 2026-10-30T04:12:13.368743Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T04:12:13.368767Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T04:14:04.904325Z info ads ADS: "" 4 terminated 2026-10-30T04:14:28.940651Z info ads ADS: "" 3 terminated with stream closed 2026-10-30T04:19:26.100105Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:21:20.245982Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:22:10.793945Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-10-30T04:24:02.558379Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:25:41.163405Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:33:32.029108Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-10-30T04:34:38.846725Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:35:03.504757Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:35:35.854049Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:35:40.575Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 147 412 96 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "5fec2d54-7a2f-4d95-853d-a7394f484213" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.31:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:44898 10.133.0.40:443 10.132.0.20:47248 - v1-chat-completions-path [2026-10-30T04:35:40.970Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 5518 5791 59 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "9551cbee-de63-4e35-8f53-c425148d57cf" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.31:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:44898 10.133.0.40:443 10.132.0.20:47252 - v1-chat-completions-path 2026-10-30T04:35:53.289985Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:35:55.751452Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:36:06.697601Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:36:24.746138Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:36:26.977953Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:36:41.758586Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:36:52.528131Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:36:54.504208Z info ads ADS: new connection for node:5 2026-10-30T04:36:54.504397Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T04:36:54.504427Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T04:36:56.202865Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:37:36.553488Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:37:46.206Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 336 84 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "7e5bb623-bf15-4ed2-86a1-db4d60d00849" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.1.10:8000" outbound|54321||e2e-prio-pool-89562748-inference-pool-ip-eef2bbad.llm.svc.cluster.local 10.133.0.40:34054 10.133.0.40:443 10.132.0.20:51536 - v1-model-routing 2026-10-30T04:38:00.712640Z info ads ADS: "" 5 terminated 2026-10-30T04:38:00.762561Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:38:02.210199Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:38:04.391885Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:38:09.457955Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:39:40.004176Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:39:40.004212Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:39:40.004227Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T04:39:40.004232Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T04:39:39.925Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 78 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "15931f5e-b67e-4171-b081-0bea4fd4347b" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.40:48294 10.133.0.40:443 10.132.0.20:34598 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:39:45.902823Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:40:02.994015Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:40:21.643157Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-30T04:40:21.649331Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:40:21.649361Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:40:21.649375Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:40:21.649380Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T04:40:21.565Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 80 322 83 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "5ae53ce9-c0d5-4990-8d3f-1cec557d9f1a" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.40:48294 10.133.0.40:443 10.132.0.20:60670 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:40:23.842765Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:40:35.832686Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:40:39.476922Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:40:49.709860Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:40:54.111830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:41:12.284879Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:41:56.234498Z info ads ADS: new connection for node:6 2026-10-30T04:41:56.240091Z info cache read certificate from file resource=file-root:system 2026-10-30T04:41:57.362562Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:41:58.914877Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:41:59.665177Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:41:59.665213Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:41:59.665228Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T04:41:59.665232Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T04:41:59.556Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 108 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "8e8b451b-f3ce-465d-8b87-2c42ea7e2787" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.40:38702 10.133.0.40:443 10.132.0.20:49684 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:42:01.820334Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"1109","StreamId":"4433067447574603473"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-10-30T04:42:01.748Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 72 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "3eb895e1-d09d-46eb-aa05-b69252cdcd4c" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.40:443 10.132.0.20:49698 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T04:42:08.105079Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"1113","StreamId":"10432367753615032802"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-10-30T04:42:08.048Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 56 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "2d42f10c-5a88-4d68-92fd-5ea7508026b8" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.40:443 10.132.0.20:46546 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T04:42:12.341906Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:42:12.341939Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:42:12.341950Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:42:12.341955Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T04:42:12.342098Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T04:42:12.229Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 112 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "b476336d-9850-4b24-b6fa-34cd8bfab60a" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.40:36210 10.133.0.40:443 10.132.0.20:46548 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T04:42:16.594676Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:42:16.594708Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:42:16.594719Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T04:42:16.594724Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:42:16.594875Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T04:42:16.487Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1732 107 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "c2bfcd8c-77f8-4f5e-95b4-a017d38e8d16" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.40:37928 10.133.0.40:443 10.132.0.20:46558 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T04:42:20.901607Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:42:20.901636Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:42:20.901645Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:42:20.901650Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T04:42:20.901787Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T04:42:20.804Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1731 96 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "bf47ed13-8d18-4f26-8eb0-fc50e95a9efc" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.40:37928 10.133.0.40:443 10.132.0.20:36036 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T04:42:25.212701Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:42:25.212736Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:42:25.212749Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:42:25.212755Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T04:42:25.212921Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T04:42:25.119Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1726 93 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "1a2b6d3a-cbc2-4f00-8ebb-f6546cfc71d5" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.40:36210 10.133.0.40:443 10.132.0.20:36060 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T04:42:52.977285Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:42:52.977319Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:42:52.977330Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T04:42:52.977335Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:42:52.977496Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T04:42:52.876Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1729 100 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e624c984-d955-41d3-8256-b0b2a8c94f90" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.40:36210 10.133.0.40:443 10.132.0.20:34840 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T04:43:05.735059Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:43:05.735093Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:43:05.735105Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:43:05.735110Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T04:43:05.735258Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T04:43:05.637Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 97 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "b3933e18-6cf5-4677-bf6f-51a29289a37b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.40:37928 10.133.0.40:443 10.132.0.20:44596 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T04:43:29.647128Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:43:29.740401Z info ads ADS: "" 6 terminated with stream closed 2026-10-30T04:43:31.020755Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:43:36.620377Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:43:55.524363Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:44:00.113385Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:44:15.451667Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:44:17.769554Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:44:20.685229Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:44:30.919788Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:44:37.179Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 72 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "ee2f82b2-e741-437b-bd23-4f1044b2eb10" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:33444 10.133.0.40:443 10.132.0.20:54478 - v1-completions-path [2026-10-30T04:44:37.279Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 57 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "9660f747-efeb-432f-80cc-708acbb43462" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:33444 10.133.0.40:443 10.132.0.20:54484 - v1-completions-path [2026-10-30T04:44:37.722Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 60 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "c8363011-36c9-4d0e-902e-b732e7519843" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:33456 10.133.0.40:443 10.132.0.20:48486 - v1-completions-path 2026-10-30T04:44:54.411738Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:45:04.678569Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:45:09.114266Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:45:26.337639Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:45:28.683387Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:45:47.799856Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:45:52.194831Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:46:07.700479Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:46:10.013376Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:46:27.670955Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:46:38.352208Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:46:41.183750Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:47:00.365894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:47:03.312628Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:47:08.966772Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:47:08.966808Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:47:08.966822Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:47:08.966827Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T04:47:08.894Z] "DELETE /v1/api-keys/8d99d2ac-4154-4b8f-9ab8-7434e01f136e HTTP/1.1" 200 - via_upstream - "-" 0 375 72 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "cc3f9855-f240-418e-88a4-85fdd2f8582e" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.40:55724 10.133.0.40:443 10.132.0.20:44208 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:47:15.590091Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:47:33.386681Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:47:43.699876Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:48:03.030696Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:48:06.360076Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:49:48.038585Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:49:51.432121Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:51:00.361204Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T04:51:00.361229Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T04:51:00.361232Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T04:51:00.361236Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T04:51:00.315Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 403 - - "-" 73 113 46 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "11a7f3c1-9919-48bd-afda-43e3d26449a6" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.40:443 10.132.0.20:53626 - v1-completions-path 2026-10-30T04:51:21.497927Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:51:24.024928Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:51:24.024961Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:51:24.024974Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T04:51:24.024979Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T04:51:23.850Z] "DELETE /v1/api-keys/e2e-trlp-test-key HTTP/1.1" 404 - via_upstream - "-" 0 29 174 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "3f4e361f-3213-4a67-a42e-0ad31005d4a5" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.40:57150 10.133.0.40:443 10.132.0.20:42094 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:51:35.951878Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:51:38.318980Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:51:44.737573Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:51:44.737611Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:51:44.737630Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:51:44.737637Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T04:51:44.661Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 65 308 76 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "3a9c4239-4263-446c-b4d7-0c788f26ec00" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.40:60496 10.133.0.40:443 10.132.0.20:58782 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:51:56.905956Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:52:08.467820Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:52:12.199708Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:52:16.098Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 57 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "08857b9b-6eac-4613-a615-fc42057810b5" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:33456 10.133.0.40:443 10.132.0.20:42042 - v1-completions-path 2026-10-30T04:52:27.384520Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:52:37.972782Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:52:40.542557Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:53:00.727441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:53:05.269824Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:53:08.213581Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:53:14.820Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 62 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "0f817335-f8fb-4c96-b092-e71ece6978d0" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:33456 10.133.0.40:443 10.132.0.20:59026 - v1-completions-path [2026-10-30T04:53:17.969Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 72 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "ea86c3a8-7234-4044-9bc2-054cb4630454" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:33444 10.133.0.40:443 10.132.0.20:52026 - v1-completions-path 2026-10-30T04:53:31.080394Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:53:41.492466Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:53:45.952221Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:53:49.001594Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:53:54.002221Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:53:59.218Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 60 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "bd36ff68-0263-45b2-a7f0-4478039249b6" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.40:33456 10.133.0.40:443 10.132.0.20:38180 - v1-completions-path 2026-10-30T04:54:12.298952Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:54:22.479979Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:54:24.795147Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:54:34.254520Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:54:39.079347Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:54:55.520925Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:15.323712Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:26.572166Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:46.182705Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:56.642184Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:56:06.858339Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:56:18.136783Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:56:28.445941Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:56:38.797248Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:56:49.127900Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:56:50.981604Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:56:50.981637Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:56:50.981651Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T04:56:50.981656Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T04:56:50.883Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 324 98 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "8ec07a63-095c-42aa-80a6-ebdd0a2b802d" "maas.apps.d36a3d177388305ff350.prod.konflux-ocp-ci.dev" "10.134.1.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.40:53790 10.133.0.40:443 10.132.0.20:45156 - odh-ai-gateway-infra.maas-api-route.3