2026-11-03T13:29:26.560373Z info FLAG: --concurrency="0" 2026-11-03T13:29:26.560392Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T13:29:26.560397Z info FLAG: --help="false" 2026-11-03T13:29:26.560401Z info FLAG: --log_as_json="false" 2026-11-03T13:29:26.560404Z info FLAG: --log_caller="" 2026-11-03T13:29:26.560407Z info FLAG: --log_output_level="default:info" 2026-11-03T13:29:26.560410Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T13:29:26.560421Z info FLAG: --log_target="[stdout]" 2026-11-03T13:29:26.560425Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T13:29:26.560427Z info FLAG: --outlierLogPath="" 2026-11-03T13:29:26.560431Z info FLAG: --profiling="true" 2026-11-03T13:29:26.560433Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T13:29:26.560436Z info FLAG: --proxyLogLevel="warning" 2026-11-03T13:29:26.560439Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T13:29:26.560442Z info FLAG: --stsPort="0" 2026-11-03T13:29:26.560445Z info FLAG: --templateFile="" 2026-11-03T13:29:26.560448Z info FLAG: --tokenManagerPlugin="" 2026-11-03T13:29:26.560451Z info FLAG: --vklog="0" 2026-11-03T13:29:26.560455Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T13:29:26.560634Z info Proxy role ips=[10.134.0.173] type=router id=maas-default-gateway-openshift-default-84587756dc-xs5rk.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T13:29:26.560696Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T13:29:26.561955Z info cpu limit detected as 2, setting concurrency 2026-11-03T13:29:26.562251Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T13:29:26.562264Z info JWT policy is third-party-jwt 2026-11-03T13:29:26.562269Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T13:29:26.563836Z info Starting default Istio SDS Server 2026-11-03T13:29:26.563866Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T13:29:26.563907Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T13:29:26.564270Z info Opening status port 15020 2026-11-03T13:29:26.565204Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T13:29:26.567405Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T13:29:26.568647Z info Starting proxy agent 2026-11-03T13:29:26.568708Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T13:29:26.576496Z info sds Starting SDS grpc server 2026-11-03T13:29:26.576515Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T13:29:26.641611Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T13:29:26.669934Z info cache generated new workload certificate resourceName=default latency=104.861016ms ttl=23h59m59.330069979s 2026-11-03T13:29:26.669988Z info cache Root cert has changed, start rotating root cert 2026-11-03T13:29:26.670032Z info cache returned workload trust anchor from cache ttl=23h59m59.329968693s 2026-11-03T13:29:26.807715Z info ads ADS: new connection for node:1 2026-11-03T13:29:26.815154Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T13:29:26.815230Z info cache read certificate from file resource=file-root:system 2026-11-03T13:29:26.845471Z info ads ADS: new connection for node:2 2026-11-03T13:29:26.845550Z info cache returned workload certificate from cache ttl=23h59m59.154451344s 2026-11-03T13:29:26.846480Z info ads ADS: new connection for node:3 2026-11-03T13:29:26.846540Z info cache returned workload trust anchor from cache ttl=23h59m59.153461111s 2026-11-03T13:29:26.948943Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:29:35.523934Z info ads ADS: new connection for node:4 2026-11-03T13:29:35.524086Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T13:29:35.524110Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T13:29:35.529845Z info Readiness succeeded in 8.972091622s 2026-11-03T13:29:35.530243Z info Envoy proxy is ready 2026-11-03T13:29:56.539595Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:30:36.060057Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:31:04.260480Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:31:31.943620Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:31:36.710941Z info ads ADS: "" 4 terminated with stream closed 2026-11-03T13:31:55.565933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:32:02.059348Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T13:32:02.059382Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T13:32:02.059392Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T13:32:02.059396Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T13:32:02.059550Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T13:32:01.823Z] "POST /llm/e2e-rt-sw-2ece70/anything/e2e-rt-sw-2ece70-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 235 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "8471ece8-1b86-4a12-8694-5b229a1618e5" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-2ece70.llm.svc.cluster.local - 10.134.0.173:443 10.132.0.13:42214 - llm.external-model-e2e-rt-sw-2ece70-8a582f45b424.2 2026-11-03T13:32:12.204524Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:32:30.084320Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:32:30.084353Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:32:30.084365Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:32:30.084370Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T13:32:30.084567Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T13:32:29.861Z] "POST /anything/e2e-rt-sw-2ece70-client HTTP/1.1" 200 - via_upstream - "-" 124 1764 222 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "21a67dd6-b263-4746-ab84-3f02640b4a29" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.173:56582 10.134.0.173:443 10.132.0.13:35322 - llm.external-model-e2e-rt-sw-2ece70-8a582f45b424.0 2026-11-03T13:32:36.733353Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:33:03.206531Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:34:41.999678Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:34:46.817185Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:34:46.817241Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:34:46.817258Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T13:34:46.817264Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:34:46.817497Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T13:34:46.694Z] "POST /anything/e2e-rt-eq-86abdb-client HTTP/1.1" 200 - via_upstream - "-" 124 1765 123 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "a095dfec-e620-4178-a60e-f96e49238d05" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.173:56582 10.134.0.173:443 10.132.0.13:48794 - llm.external-model-e2e-rt-eq-86abdb-c702f89b7e8b.0 2026-11-03T13:34:48.434284Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:34:48.434314Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:34:48.434325Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T13:34:48.434330Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:34:48.434492Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T13:34:48.324Z] "POST /anything/e2e-rt-eq-86abdb-client HTTP/1.1" 200 - via_upstream - "-" 124 1759 110 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "e933aaa4-7306-443f-ac86-6ade16967217" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.173:56582 10.134.0.173:443 10.132.0.13:48820 - llm.external-model-e2e-rt-eq-86abdb-c702f89b7e8b.2 2026-11-03T13:34:49.632130Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-03T13:34:49.632811Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:34:49.632835Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:34:49.632845Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:34:49.632850Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T13:34:49.632994Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T13:34:49.419Z] "POST /anything/e2e-rt-eq-86abdb-client HTTP/1.1" 200 - via_upstream - "-" 124 1765 213 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "742568e9-4937-4412-8a76-03763a2b734d" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.173:56582 10.134.0.173:443 10.132.0.13:48830 - llm.external-model-e2e-rt-eq-86abdb-c702f89b7e8b.0 2026-11-03T13:35:14.993191Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:35:31.513749Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:35:53.378515Z info ads ADS: "" 1 terminated 2026-11-03T13:36:34.497297Z info ads ADS: new connection for node:5 2026-11-03T13:36:34.526885Z info cache read certificate from file resource=file-root:system 2026-11-03T13:37:18.861057Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:38:08.125560Z info ads ADS: "" 5 terminated with stream closed 2026-11-03T13:39:04.866468Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:42:23.348433Z info ads ADS: new connection for node:6 2026-11-03T13:42:23.375455Z info cache read certificate from file resource=file-root:system 2026-11-03T13:44:21.147868Z info ads ADS: "" 6 terminated 2026-11-03T13:46:07.374646Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:46:07.374695Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:46:07.374719Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:46:07.374725Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T13:46:07.290Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 30 315 83 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "c49188fc-987c-4eb4-9255-7981bc14acd1" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.133.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.173:42138 10.134.0.173:443 10.132.0.13:59188 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T13:46:09.459654Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:46:20.247308Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-03T13:46:33.904974Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:48:35.699933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:48:50.930973Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:48:52.648331Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:49:02.576221Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:49:05.909277Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:49:09.421535Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:49:24.717047Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:49:44.648259Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:49:54.880467Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:50:05.190808Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:50:15.517118Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:50:25.633945Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:50:35.869351Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:50:46.090739Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:50:56.276722Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:51:06.418111Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:51:16.683410Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:51:54.112508Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T13:51:57.432Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 2817 3089 81 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "1ecf38a3-c803-4b0c-bac5-04e951e0d7a3" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.42:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:39006 10.134.0.173:443 10.132.0.13:56914 - v1-chat-completions-path 2026-11-03T13:52:10.044462Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:52:11.717163Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:52:15.060912Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T13:52:14.981Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 78 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "1c461585-2b44-4d89-a0b2-77f8afab63c9" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.134.0.31:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:47980 10.134.0.173:443 10.132.0.13:43090 - v1-catch-all-path 2026-11-03T13:52:17.515689Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:52:33.017875Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:52:34.634967Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:52:37.676144Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T13:52:37.609Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 67 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "b1fe255a-0c15-4e36-b45e-256a4c67cb3a" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.134.0.31:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:47980 10.134.0.173:443 10.132.0.13:34214 - v1-catch-all-path 2026-11-03T13:52:49.213048Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:52:59.939837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:53:01.678313Z info ads ADS: new connection for node:7 2026-11-03T13:53:01.678570Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T13:53:02.886994Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:53:43.804880Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:54:02.787598Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:54:03.295284Z info ads ADS: "" 7 terminated with stream closed 2026-11-03T13:54:04.580966Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:54:07.706475Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:54:11.667295Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:54:23.789640Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:54:40.454550Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:54:56.754495Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:55:08.381635Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:55:08.711865Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T13:55:08.711914Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T13:55:08.711935Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T13:55:08.711943Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T13:55:08.584Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 84 326 127 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "38e5855a-5c13-4d31-ab79-71af6ae556e8" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.133.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.173:58978 10.134.0.173:443 10.132.0.13:42846 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T13:55:12.198644Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:55:20.340119Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:55:22.937850Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:55:38.586494Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T13:56:57.130Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 110 321 76 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "fa3e517a-07ac-4691-a462-20e3465a04b0" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:41126 10.134.0.173:443 10.132.0.13:34598 - v1-chat-completions-path 2026-11-03T13:59:40.760809Z info ads ADS: new connection for node:8 2026-11-03T13:59:40.769955Z info cache read certificate from file resource=file-root:system 2026-11-03T13:59:41.478455Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:59:42.899143Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T13:59:46.124700Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:59:46.124744Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:59:46.124761Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T13:59:46.124766Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T13:59:46.270675Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"504","StreamId":"16170394363231951870"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-03T13:59:45.487Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 637 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "b22ee8ff-f557-4f36-9033-09a7e186a5f2" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.133.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.173:39114 10.134.0.173:443 10.132.0.13:36884 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T13:59:46.182Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 88 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "309b60d5-b65a-432e-a200-1e36e9dcff25" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.134.0.173:443 10.132.0.13:48232 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T14:00:10.237619Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T14:00:10.237655Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T14:00:10.237666Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T14:00:10.237671Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T14:00:10.237830Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T14:00:10.111Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1715 126 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "7c918425-fdfe-43a8-9dde-d3a406a90c09" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.173:38108 10.134.0.173:443 10.132.0.13:42902 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T14:00:12.415450Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T14:00:12.415486Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T14:00:12.415497Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T14:00:12.415501Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T14:00:12.415661Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T14:00:12.293Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1714 121 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "55f919f5-2a39-4de2-b85c-5c800f89a1a5" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.173:38122 10.134.0.173:443 10.132.0.13:42912 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T14:00:36.151424Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T14:00:36.255912Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T14:00:36.255946Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T14:00:36.255960Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T14:00:36.255965Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T14:00:36.256159Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T14:00:36.144Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1716 111 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "d03bc750-becf-416f-9d25-f4243cf48258" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.173:38122 10.134.0.173:443 10.132.0.13:43296 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T14:00:40.574684Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T14:00:40.574728Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T14:00:40.574744Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T14:00:40.574751Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T14:00:40.574977Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T14:00:40.476Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1720 98 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "138b8ff1-bf9f-403b-a3f6-269f7a1bc380" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.173:38108 10.134.0.173:443 10.132.0.13:43302 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T14:01:26.675116Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:01:26.792327Z info ads ADS: "" 8 terminated 2026-11-03T14:01:28.362618Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:01:31.419837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:01:43.409467Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:01:45.263292Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:01:47.939050Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:01:56.619795Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T14:02:02.779Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 76 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "293f56c3-c244-4706-b5f6-10a5bde337ec" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:56802 - v1-completions-path [2026-11-03T14:02:04.191Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 60 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "68f7c146-f701-4fa6-9219-fd115af3c2fe" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46022 10.134.0.173:443 10.132.0.13:56810 - v1-completions-path [2026-11-03T14:02:04.795Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 53 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "2205e027-4bde-492e-a215-dedee3e629bb" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:51076 - v1-completions-path [2026-11-03T14:02:05.233Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 58 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "c491cc0d-1d0b-4ab9-8b04-a12b6d399bcf" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:51078 - v1-completions-path [2026-11-03T14:02:05.477Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 60 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "50758cf4-1b71-482f-ae58-a4b3557c5785" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:51080 - v1-completions-path 2026-11-03T14:02:21.382096Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:02:31.661341Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:02:33.291512Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T14:02:39.513Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "23835bea-fc8c-4b35-a81e-e580ea2768ba" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46022 10.134.0.173:443 10.132.0.13:44290 - v1-completions-path [2026-11-03T14:02:39.853Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 55 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "b534b58e-f5c3-4bce-91be-542a0ee16a6d" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:44300 - v1-completions-path [2026-11-03T14:02:39.961Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 56 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "0c93eae1-4876-4ca0-bbdd-7d5d3350334b" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:44306 - v1-completions-path [2026-11-03T14:02:40.308Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 57 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "59853bb6-8eea-4d60-806d-59de5972d998" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46022 10.134.0.173:443 10.132.0.13:44312 - v1-completions-path 2026-11-03T14:02:52.306158Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:02:54.012167Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:02:57.118600Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T14:02:57.118637Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T14:02:57.118656Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T14:02:57.118661Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T14:02:56.685Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 433 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "300dc297-e29e-4e29-9877-b0ece2e7cef8" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.133.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.173:54464 10.134.0.173:443 10.132.0.13:55646 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T14:02:59.865168Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:03:10.793727Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:03:24.465721Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:03:26.052343Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:03:31.708538Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:03:49.817254Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:03:52.388162Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:04:05.200785Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:04:07.296742Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:04:11.806902Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:04:30.160278Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:04:40.395621Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:04:56.873098Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:04:59.836788Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:07:14.442242Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:07:17.474888Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:07:19.663161Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:07:34.814481Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:07:45.012381Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:08:04.972871Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:11:40.701328Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:11:45.123441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T14:11:48.350Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "2177f45e-9dc5-43cd-b333-16591d4430d6" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:40950 - v1-completions-path [2026-11-03T14:11:49.400Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 62 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "7c23625a-1085-403d-ab68-16e6c2318bf0" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:40962 - v1-completions-path 2026-11-03T14:12:01.284908Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:12:11.406309Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:12:14.741446Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:12:20.261053Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T14:12:20.261092Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T14:12:20.261109Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T14:12:20.261114Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T14:12:20.196Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 309 64 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "9e7947f1-47d2-4dc0-865e-028c77fb357b" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.134.1.28:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.173:46856 10.134.0.173:443 10.132.0.13:46408 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T14:12:20.319Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 63 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "3e54bb10-7746-4ae2-a371-6ae0b33230bd" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46022 10.134.0.173:443 10.132.0.13:46420 - v1-completions-path [2026-11-03T14:12:20.800Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 56 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "4afb4a99-5245-47aa-b848-5d404d6c719d" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46022 10.134.0.173:443 10.132.0.13:46422 - v1-completions-path [2026-11-03T14:12:21.228Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 60 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "fdd97ef2-437d-4c4a-a094-cfcfb1c29e5c" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46022 10.134.0.173:443 10.132.0.13:46438 - v1-completions-path 2026-11-03T14:12:33.358240Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:12:35.510856Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:12:38.807635Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T14:12:43.226Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 61 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "bf10e1c4-f71e-4bf0-bbe6-11ac8fe4e334" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:46634 - v1-completions-path [2026-11-03T14:12:43.564Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 59 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "7831f31b-262e-45a9-9196-3a79b18e94de" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46022 10.134.0.173:443 10.132.0.13:46664 - v1-completions-path [2026-11-03T14:12:44.207Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 56 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "45b8cdce-e0d6-41e5-9a0c-19cff6647208" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:46666 - v1-completions-path [2026-11-03T14:12:44.693Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 56 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "db200f74-e8b3-4ba0-a042-48651909c923" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:51808 - v1-completions-path [2026-11-03T14:12:47.180Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 58 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "d49f7c96-e4d0-482c-b3e6-773cc7be0428" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46022 10.134.0.173:443 10.132.0.13:51818 - v1-completions-path 2026-11-03T14:12:59.611147Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:13:09.898735Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:13:12.082240Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:13:15.427319Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:13:23.523995Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-03T14:13:29.811Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 62 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "f2bbdd7a-4987-4921-8a05-cb964015a69c" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.173:46008 10.134.0.173:443 10.132.0.13:45172 - v1-completions-path 2026-11-03T14:13:42.221197Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:13:52.412336Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-03T14:13:53.209150Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T14:13:53.209177Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T14:13:53.209180Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T14:13:53.209184Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-03T14:13:53.187Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 22 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "ced2ec2c-5a3b-4b94-a46b-5c508409e5c4" "maas.apps.013b5c78a9b24bb4ebb6.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.173:443 10.132.0.13:60244 - v1-completions-path