2026-10-31T04:13:27.032104Z info FLAG: --concurrency="0" 2026-10-31T04:13:27.032131Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-31T04:13:27.032135Z info FLAG: --help="false" 2026-10-31T04:13:27.032138Z info FLAG: --log_as_json="false" 2026-10-31T04:13:27.032140Z info FLAG: --log_caller="" 2026-10-31T04:13:27.032142Z info FLAG: --log_output_level="default:info" 2026-10-31T04:13:27.032144Z info FLAG: --log_stacktrace_level="default:none" 2026-10-31T04:13:27.032148Z info FLAG: --log_target="[stdout]" 2026-10-31T04:13:27.032150Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-31T04:13:27.032166Z info FLAG: --outlierLogPath="" 2026-10-31T04:13:27.032169Z info FLAG: --profiling="true" 2026-10-31T04:13:27.032171Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-31T04:13:27.032173Z info FLAG: --proxyLogLevel="warning" 2026-10-31T04:13:27.032175Z info FLAG: --serviceCluster="istio-proxy" 2026-10-31T04:13:27.032177Z info FLAG: --stsPort="0" 2026-10-31T04:13:27.032179Z info FLAG: --templateFile="" 2026-10-31T04:13:27.032180Z info FLAG: --tokenManagerPlugin="" 2026-10-31T04:13:27.032183Z info FLAG: --vklog="0" 2026-10-31T04:13:27.032186Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-31T04:13:27.032400Z info Proxy role ips=[10.134.0.42] type=router id=maas-default-gateway-openshift-default-77bfb58f8c-wzfdx.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-31T04:13:27.032445Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-31T04:13:27.033505Z info cpu limit detected as 2, setting concurrency 2026-10-31T04:13:27.033692Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-31T04:13:27.033703Z info JWT policy is third-party-jwt 2026-10-31T04:13:27.033706Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-31T04:13:27.034943Z info Starting default Istio SDS Server 2026-10-31T04:13:27.034964Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-31T04:13:27.034988Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-31T04:13:27.035438Z info Opening status port 15020 2026-10-31T04:13:27.036407Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-31T04:13:27.037784Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-31T04:13:27.038656Z info Starting proxy agent 2026-10-31T04:13:27.038692Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-31T04:13:27.047928Z info sds Starting SDS grpc server 2026-10-31T04:13:27.047953Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-31T04:13:27.102631Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-31T04:13:27.149593Z info cache generated new workload certificate resourceName=default latency=112.63531ms ttl=23h59m59.850411532s 2026-10-31T04:13:27.149629Z info cache Root cert has changed, start rotating root cert 2026-10-31T04:13:27.149660Z info cache returned workload trust anchor from cache ttl=23h59m59.850340205s 2026-10-31T04:13:27.243479Z info ads ADS: new connection for node:1 2026-10-31T04:13:27.243543Z info cache returned workload certificate from cache ttl=23h59m59.756458757s 2026-10-31T04:13:27.243806Z info ads ADS: new connection for node:2 2026-10-31T04:13:27.243905Z info cache returned workload trust anchor from cache ttl=23h59m59.756097179s 2026-10-31T04:13:27.324697Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:13:35.129986Z info Readiness succeeded in 8.100214358s 2026-10-31T04:13:35.130327Z info Envoy proxy is ready 2026-10-31T04:17:03.886785Z info ads ADS: new connection for node:3 2026-10-31T04:17:03.892390Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-31T04:17:03.892440Z info cache read certificate from file resource=file-root:system 2026-10-31T04:17:05.042519Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:17:11.187424Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:17:20.070012Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:17:24.391588Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:17:24.391633Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:17:24.391647Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:17:24.391653Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T04:17:24.391903Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T04:17:24.260Z] "POST /llm/e2e-rt-sw-c6819e/anything/e2e-rt-sw-c6819e-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 131 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "713b2efb-d4e3-4765-bf49-544180201b62" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-c6819e.llm.svc.cluster.local - 10.134.0.42:443 10.132.0.18:46568 - llm.external-model-e2e-rt-sw-c6819e-1a780d48fc44.2 2026-10-31T04:17:32.605491Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:17:42.796960Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:17:42.797011Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:17:42.797027Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T04:17:42.797034Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:17:42.797286Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T04:17:42.671Z] "POST /anything/e2e-rt-sw-c6819e-client HTTP/1.1" 200 - via_upstream - "-" 124 1774 126 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "7a084dc4-22b2-4d8a-8429-ba99bda9cd6b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.42:46646 10.134.0.42:443 10.132.0.18:38136 - llm.external-model-e2e-rt-sw-c6819e-1a780d48fc44.0 2026-10-31T04:17:49.171147Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:18:04.161392Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:18:04.161423Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:18:04.161433Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:18:04.161439Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T04:18:04.161593Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T04:18:04.047Z] "POST /anything/e2e-rt-sw-c6819e-client HTTP/1.1" 200 - via_upstream - "-" 124 1767 114 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "147bf263-3bc5-4eda-9e2b-6b7e0b637cf5" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.42:50068 10.134.0.42:443 10.132.0.18:32946 - llm.external-model-e2e-rt-sw-c6819e-1a780d48fc44.0 2026-10-31T04:18:13.761863Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:18:28.529588Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:18:57.008370Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:18:57.072885Z info ads ADS: "" 3 terminated 2026-10-31T04:18:57.696752Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:19:12.397270Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:20:09.701906Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:20:39.614383Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:21:28.119754Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:21:28.119792Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:21:28.119807Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T04:21:28.119812Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:21:28.759752Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:21:28.759799Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:21:28.759819Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T04:21:28.759826Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T04:21:27.888Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 231 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "30d2b8a4-f446-4cee-a630-77106a853f4b" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:37618 10.134.0.42:443 10.132.0.18:45646 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T04:21:28.663Z] "DELETE /v1/api-keys/cb4cea64-f98e-4103-808e-e4ebeaa7f5d4 HTTP/1.1" 200 - via_upstream - "-" 0 321 96 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "d67f6f13-d7ae-4de3-a256-49b311283d71" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:37618 10.134.0.42:443 10.132.0.18:45658 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T04:21:28.783Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 56 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "e670baf9-ef6e-4ffa-b64a-fcddb8e4d499" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.42:443 10.132.0.18:45670 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T04:21:30.806196Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:21:30.806246Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:21:30.806268Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T04:21:30.806275Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T04:21:30.717Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 88 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "818f54f4-938f-4e82-ba53-81c29ea3bfb1" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:37618 10.134.0.42:443 10.132.0.18:45682 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T04:21:31.871430Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:21:31.871470Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:21:31.871489Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:21:31.871496Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-31T04:21:31.797Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 73 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "41617a6e-96fe-4c26-b8c2-76722d7ea8d2" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:46806 10.134.0.42:443 10.132.0.18:45686 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T04:22:15.789226Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:22:53.978752Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:22:53.978790Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:22:53.978807Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T04:22:53.978812Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-31T04:22:53.896Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 82 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "7dccb844-599e-4325-a21c-bb196afb9fce" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:47594 10.134.0.42:443 10.132.0.18:58076 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T04:29:21.351085Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-10-31T04:29:21.274Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 135 2570 97 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "e9d37098-a039-411a-8f66-934815490a91" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:47988 10.134.0.42:443 10.132.0.18:53308 - v1-chat-completions-path 2026-10-31T04:41:34.775304Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-31T04:41:37.186614Z info ads ADS: new connection for node:4 2026-10-31T04:41:37.192170Z info cache read certificate from file resource=file-root:system 2026-10-31T04:47:55.912463Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:48:24.364981Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:48:31.100509Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:48:31.100551Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:48:31.100574Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:48:31.100581Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T04:48:31.274916Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:48:31.274953Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:48:31.274968Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T04:48:31.274974Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T04:48:31.275195Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T04:48:30.968Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 66 309 131 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "385daf26-4fd4-45ce-9675-9f7d51062cc2" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:38864 10.134.0.42:443 10.132.0.18:53346 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T04:48:31.129Z] "POST /anything/e2e-dft-5dcd93-b90afd HTTP/1.1" 200 - via_upstream - "-" 109 1770 145 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "50c8c28c-1841-4900-bc48-073b8b12d350" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.42:34212 10.134.0.42:443 10.132.0.18:53360 - llm.external-model-e2e-dft-5dcd93-285f860ea43f.0 2026-10-31T04:48:44.223321Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:48:47.611808Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:48:52.203549Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:48:59.429536Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:49:05.538679Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:49:08.499052Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:49:13.648446Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:49:15.025782Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:50:34.758699Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:50:53.662390Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:51:13.170000Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:51:33.962432Z info ads ADS: new connection for node:5 2026-10-31T04:51:33.962635Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-31T04:51:33.962665Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-31T04:51:37.989854Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:51:40.598595Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:52:17.664456Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:53:09.710083Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T04:53:15.808Z] "POST /llm/e2e-dft-8b11d9/anything/e2e-dft-8b11d9-9a98c3 HTTP/1.1" 403 - - "-" 109 77 61 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "a6f43bbc-a195-444e-9963-a471e0c71c4a" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-8b11d9.llm.svc.cluster.local - 10.134.0.42:443 10.132.0.18:35850 - llm.external-model-e2e-dft-8b11d9-d604eae12064.2 2026-10-31T04:53:21.178600Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-10-31T04:53:21.178679Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:53:21.178695Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:53:21.178705Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T04:53:21.178709Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T04:53:21.178860Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T04:53:21.066Z] "POST /anything/e2e-dft-8b11d9-9a98c3 HTTP/1.1" 200 - via_upstream - "-" 109 1773 112 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "d8e7e66e-5726-4189-8221-6075e56f6897" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.42:44212 10.134.0.42:443 10.132.0.18:35854 - llm.external-model-e2e-dft-8b11d9-d604eae12064.0 2026-10-31T04:53:33.422105Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:54:14.530293Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T04:55:33.657475Z info ads ADS: "" 5 terminated 2026-10-31T04:55:35.736360Z info ads ADS: "" 4 terminated with stream closed 2026-10-31T05:01:29.237538Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-10-31T05:05:36.638058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:07:40.279169Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:08:00.992048Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:09:43.651240Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:13:56.504254Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-10-31T05:18:55.009633Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:19:17.928880Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:19:29.137498Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:19:29.137531Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:19:29.137543Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:19:29.137548Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T05:19:28.967Z] "POST /v1/api-keys HTTP/1.1" 400 - via_upstream - "-" 80 94 170 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "7dd3b0d4-6d88-4fa5-840d-c2fed0f255f4" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:33522 10.134.0.42:443 10.132.0.18:48208 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T05:19:50.208745Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:20:06.488428Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:20:09.965538Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:20:13.110634Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-10-31T05:20:12.297Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 812 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "f1788c9a-8ae5-4eca-9b2f-10b0ef9a87b2" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:49246 10.134.0.42:443 10.132.0.18:40080 - v1-catch-all-path 2026-10-31T05:20:16.110015Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:20:24.163726Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:20:33.656274Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:20:48.842878Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:20:59.834370Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:21:02.200798Z info ads ADS: new connection for node:6 2026-10-31T05:21:02.200955Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-31T05:21:02.200979Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-31T05:21:03.972644Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:21:44.322831Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:21:47.069Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 346 658 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "857ddcae-b2a9-4f30-a59f-1937195f2c8e" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.1.226:8000" outbound|54321||e2e-prio-pool-119c987e-inference-pool-ip-e01123fa.llm.svc.cluster.local 10.134.0.42:45108 10.134.0.42:443 10.132.0.18:41510 - v1-model-routing [2026-10-31T05:21:49.092Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 336 67 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "bbf3b2f5-cc34-4724-894d-f79ab3643c0a" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.1.226:8000" outbound|54321||e2e-prio-pool-119c987e-inference-pool-ip-e01123fa.llm.svc.cluster.local 10.134.0.42:45114 10.134.0.42:443 10.132.0.18:41530 - v1-model-routing [2026-10-31T05:21:51.527Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 346 58 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "61bdc46b-beb5-4677-83af-928748b8b693" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.1.226:8000" outbound|54321||e2e-prio-pool-119c987e-inference-pool-ip-e01123fa.llm.svc.cluster.local 10.134.0.42:45108 10.134.0.42:443 10.132.0.18:41546 - v1-model-routing 2026-10-31T05:22:04.124111Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:22:05.145815Z info ads ADS: "" 6 terminated 2026-10-31T05:22:07.514120Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:22:11.480824Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:22:16.295045Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:24:34.992347Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:24:50.622713Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:25:07.296709Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:25:19.314143Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:25:24.649206Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:25:28.478244Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:25:38.789472Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:25:43.401517Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:25:58.877357Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:26:30.807Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 292 66 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "e8344fdc-1890-4592-8874-a75339e142bb" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:59118 10.134.0.42:443 10.132.0.18:46036 - v1-completions-path 2026-10-31T05:26:45.983872Z info ads ADS: new connection for node:7 2026-10-31T05:26:45.988392Z info cache read certificate from file resource=file-root:system 2026-10-31T05:26:47.047695Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:26:48.723421Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:27:31.808226Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:27:31.808260Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:27:31.808272Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T05:27:31.808277Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:27:31.808443Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T05:27:31.657Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1710 150 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "f2334b29-1580-45ca-9868-4ba222ed8205" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.42:50432 10.134.0.42:443 10.132.0.18:34050 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-31T05:27:36.059707Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T05:27:36.059744Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T05:27:36.059755Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T05:27:36.059760Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T05:27:36.059931Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T05:27:35.952Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1718 107 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "8213eab7-c4a2-429e-a0d6-4c2280ee1b1f" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.42:50434 10.134.0.42:443 10.132.0.18:42158 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-31T05:28:00.510254Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:28:00.510286Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:28:00.510298Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T05:28:00.510302Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:28:00.510490Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T05:28:00.407Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1716 103 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "bac4cd1a-2d86-432f-b799-2546f30f9fea" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.42:50432 10.134.0.42:443 10.132.0.18:55446 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-31T05:28:16.003036Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:28:16.131226Z info ads ADS: "" 7 terminated 2026-10-31T05:28:17.712025Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:28:24.991671Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:28:41.363955Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:28:46.150873Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:29:01.656115Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:29:17.396170Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:29:24.143Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 80 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "f56dd775-3261-467a-8d08-9173a7683369" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.48:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:45020 10.134.0.42:443 10.132.0.18:54920 - v1-completions-path 2026-10-31T05:29:41.034389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:29:51.532086Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:29:55.461401Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:29:59.984Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 63 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "01f4be68-e327-459c-a598-d85994f8ad5b" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.48:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:45020 10.134.0.42:443 10.132.0.18:39200 - v1-completions-path [2026-10-31T05:30:00.812Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 62 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "69e7df6d-8081-4139-a77f-7f56ce63338a" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.48:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:45020 10.134.0.42:443 10.132.0.18:39214 - v1-completions-path [2026-10-31T05:30:00.896Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 62 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "b55ff57d-7bd2-47c1-abad-b7e79605a1be" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.48:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:45020 10.134.0.42:443 10.132.0.18:39218 - v1-completions-path [2026-10-31T05:30:01.266Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 62 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "872c0552-7fe7-4c00-8dc1-aa4afc77ba8f" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.48:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:39462 10.134.0.42:443 10.132.0.18:39222 - v1-completions-path 2026-10-31T05:30:13.332808Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:30:15.718149Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:30:23.590622Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:30:35.351895Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:30:40.186325Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:30:48.786914Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:30:54.533358Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:31:12.363673Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:31:15.732057Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:31:39.535337Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:31:57.501335Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:32:07.825541Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:36:14.616922Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:36:30.340827Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:36:32.617352Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:36:35.753765Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T05:36:35.753793Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T05:36:35.753798Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T05:36:35.753802Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-31T05:36:35.117Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 636 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "f7255790-1e63-4edf-be53-2d094e9ab317" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.42:443 10.132.0.18:34140 - v1-completions-path 2026-10-31T05:36:39.916276Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:36:59.500779Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:37:02.833825Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:37:03.102Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 93 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "6f4b8ffe-9f09-42a1-9aed-7d7aae5753de" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.48:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:39462 10.134.0.42:443 10.132.0.18:53116 - v1-completions-path 2026-10-31T05:37:06.688261Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:37:06.688298Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:37:06.688312Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T05:37:06.688317Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T05:37:06.608Z] "DELETE /v1/api-keys/aecc9f5b-e020-4e22-8c74-bc97da9393e1 HTTP/1.1" 200 - via_upstream - "-" 0 369 79 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "9dc9c69b-7ac5-4984-aea1-8e2024bbbd3d" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.133.0.255:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.42:55822 10.134.0.42:443 10.132.0.18:53124 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T05:37:18.686384Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:37:29.018618Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:37:33.996073Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:37:37.778Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 67 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "5e5e52e5-bffe-4286-b16b-a4b27f600bd0" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.48:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:45020 10.134.0.42:443 10.132.0.18:37360 - v1-completions-path 2026-10-31T05:37:50.978045Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:37:55.370146Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:37:59.274834Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:38:05.802Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 70 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "558eb275-7e49-4e36-adc4-2ca0ab5778c6" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "10.132.0.48:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.42:39462 10.134.0.42:443 10.132.0.18:51738 - v1-completions-path [2026-10-31T05:38:05.998Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 58 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "b6abb7c5-531e-4cad-92b6-eda2d5c2ce66" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.42:443 10.132.0.18:51740 - v1-completions-path 2026-10-31T05:38:21.879531Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:38:32.360870Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:38:36.918186Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:38:40.845358Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:38:45.905938Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:39:04.337392Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:39:14.570512Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:39:16.864607Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:39:26.543711Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:39:30.409609Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:39:47.797572Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:39:50.036462Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:40:08.534325Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:40:18.940726Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:40:29.299252Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:40:39.602754Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:40:49.930188Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:41:00.347504Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:41:10.703270Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:41:21.108431Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:41:31.393235Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-31T05:41:41.672191Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-31T05:41:49.150Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 41 - "52.71.36.72,10.132.0.18" "python-requests/2.34.2" "32dba8b6-caf9-4871-a71a-05d07ee0ddad" "maas.apps.395278aa9995a3d74ddf.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.42:443 10.132.0.18:53904 - v1-completions-path