2026-11-02T08:24:42.033688Z info FLAG: --concurrency="0" 2026-11-02T08:24:42.033708Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T08:24:42.033712Z info FLAG: --help="false" 2026-11-02T08:24:42.033715Z info FLAG: --log_as_json="false" 2026-11-02T08:24:42.033718Z info FLAG: --log_caller="" 2026-11-02T08:24:42.033720Z info FLAG: --log_output_level="default:info" 2026-11-02T08:24:42.033722Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T08:24:42.033726Z info FLAG: --log_target="[stdout]" 2026-11-02T08:24:42.033729Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T08:24:42.033732Z info FLAG: --outlierLogPath="" 2026-11-02T08:24:42.033734Z info FLAG: --profiling="true" 2026-11-02T08:24:42.033736Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T08:24:42.033738Z info FLAG: --proxyLogLevel="warning" 2026-11-02T08:24:42.033740Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T08:24:42.033742Z info FLAG: --stsPort="0" 2026-11-02T08:24:42.033744Z info FLAG: --templateFile="" 2026-11-02T08:24:42.033746Z info FLAG: --tokenManagerPlugin="" 2026-11-02T08:24:42.033748Z info FLAG: --vklog="0" 2026-11-02T08:24:42.033751Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T08:24:42.033975Z info Proxy role ips=[10.132.1.118] type=router id=maas-default-gateway-openshift-default-7f5667f86c-jxxsm.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T08:24:42.034030Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T08:24:42.035029Z info cpu limit detected as 2, setting concurrency 2026-11-02T08:24:42.035382Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T08:24:42.035395Z info JWT policy is third-party-jwt 2026-11-02T08:24:42.035398Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T08:24:42.036861Z info Starting default Istio SDS Server 2026-11-02T08:24:42.036878Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T08:24:42.036917Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T08:24:42.037442Z info Opening status port 15020 2026-11-02T08:24:42.037828Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T08:24:42.041111Z info sds Starting SDS grpc server 2026-11-02T08:24:42.041134Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T08:24:42.041634Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T08:24:42.042802Z info Starting proxy agent 2026-11-02T08:24:42.042843Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T08:24:42.119059Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T08:24:42.208991Z info cache generated new workload certificate resourceName=default latency=170.666528ms ttl=23h59m59.791013115s 2026-11-02T08:24:42.209029Z info cache Root cert has changed, start rotating root cert 2026-11-02T08:24:42.209063Z info cache returned workload trust anchor from cache ttl=23h59m59.790937436s 2026-11-02T08:24:42.381906Z info ads ADS: new connection for node:1 2026-11-02T08:24:42.381985Z info cache returned workload certificate from cache ttl=23h59m59.618017255s 2026-11-02T08:24:42.382534Z info ads ADS: new connection for node:2 2026-11-02T08:24:42.382642Z info cache returned workload trust anchor from cache ttl=23h59m59.617358854s 2026-11-02T08:24:42.504799Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:24:50.975074Z info Readiness succeeded in 8.943756198s 2026-11-02T08:24:50.975527Z info Envoy proxy is ready 2026-11-02T08:25:41.158076Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:26:36.768559Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-02T08:28:10.446848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:30:01.559624Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:39:14.946389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:39:37.708679Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:40:10.919427Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:40:26.669441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:40:29.199973Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:40:35.665845Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:40:38.405064Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:40:38.405118Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:40:38.405143Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T08:40:38.405149Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T08:40:38.218Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 318 186 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "300337b2-f1cd-4aa4-bba8-cfa3773cf55a" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.118:40834 10.132.1.118:443 10.132.0.12:48768 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T08:40:38.712360Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-02T08:40:38.631Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 80 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "052e6cd0-168b-47bd-8b35-13a0400b59e3" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.132.0.48:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.118:58278 10.132.1.118:443 10.132.0.12:41438 - v1-catch-all-path 2026-11-02T08:40:43.253598Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:40:52.578393Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:41:07.406284Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:41:18.397626Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:41:21.025748Z info ads ADS: new connection for node:3 2026-11-02T08:41:21.026032Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T08:41:21.026131Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T08:41:23.782832Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:42:02.827869Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:42:21.813319Z info ads ADS: "" 3 terminated 2026-11-02T08:42:21.858473Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:42:25.303656Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:42:33.537953Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:44:56.336868Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:44:56.336941Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:44:56.336960Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T08:44:56.336965Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T08:44:56.246Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 90 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "7b404560-af93-4b95-be95-e8fddb03b150" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.118:35938 10.132.1.118:443 10.132.0.12:45534 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T08:45:04.149316Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:45:19.459531Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:45:35.481954Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:45:47.403528Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:45:51.061148Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:46:01.580073Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:46:06.285880Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:46:08.901285Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:46:08.901327Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:46:08.901343Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T08:46:08.901348Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T08:46:08.768Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 93 336 132 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "6d0dfb35-efe8-41c8-b064-0bf4078f0efa" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.118:56978 10.132.1.118:443 10.132.0.12:33490 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T08:46:21.581072Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:47:07.654987Z info ads ADS: new connection for node:4 2026-11-02T08:47:07.662198Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T08:47:07.662252Z info cache read certificate from file resource=file-root:system 2026-11-02T08:47:08.979364Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:47:10.601653Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:47:11.170730Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"386","StreamId":"17589025335546237364"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-02T08:47:11.158Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 12 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "64157295-738a-46a3-b926-944b1aefd72e" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.1.118:443 10.132.0.12:35568 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-02T08:47:26.150262Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:47:26.150296Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:47:26.150308Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T08:47:26.150312Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:47:26.150469Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T08:47:26.037Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1726 113 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "6042f03d-0e7d-4422-a88f-183e6bf9c1b3" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.118:54308 10.132.1.118:443 10.132.0.12:38444 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T08:47:58.079448Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:47:58.079483Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:47:58.079494Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:47:58.079498Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T08:47:58.079656Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T08:47:57.981Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1724 98 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "2b18ec63-7e71-416e-834b-b09ff5bc7307" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.118:54308 10.132.1.118:443 10.132.0.12:46950 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T08:48:24.475179Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:48:24.475213Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:48:24.475220Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-02T08:48:24.475230Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T08:48:24.475234Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:48:24.475395Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T08:48:24.365Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1718 109 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "6bbe979b-209f-4735-aee1-87a49fecfdcd" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.118:38038 10.132.1.118:443 10.132.0.12:53946 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T08:48:32.957664Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:48:32.957697Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:48:32.957708Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T08:48:32.957712Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:48:32.957855Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T08:48:32.860Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1725 97 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "a5f414e5-e87c-4294-8a3f-4a3c03eadef6" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.118:38038 10.132.1.118:443 10.132.0.12:48288 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T08:48:48.498263Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:48:48.590741Z info ads ADS: "" 4 terminated 2026-11-02T08:48:50.029672Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:48:56.787676Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:49:12.581103Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:49:17.244908Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:49:32.567869Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:49:48.327218Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:49:53.267700Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:49:53.267739Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:49:53.267753Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:49:53.267758Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T08:49:53.184Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 77 320 82 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "650b69c7-96c6-4959-8693-0e595ee3d8fd" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.118:55782 10.132.1.118:443 10.132.0.12:55744 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T08:49:54.073Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 56 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "26097f8b-f055-4680-8633-8665d8df9d71" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.118:443 10.132.0.12:55760 - v1-completions-path [2026-11-02T08:49:54.340Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 71 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "d6bc15c6-14dd-4fe4-8b5e-0321024c0931" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.118:45880 10.132.1.118:443 10.132.0.12:55772 - v1-completions-path 2026-11-02T08:50:11.395482Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:50:21.696350Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:50:26.402856Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:50:30.566023Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-02T08:50:30.016Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 63 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "a37a5662-5211-4454-9a1c-1a38d0cba2fa" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.118:45880 10.132.1.118:443 10.132.0.12:59048 - v1-completions-path [2026-11-02T08:50:30.510Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 63 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "c9c0dc9d-51a1-47cb-b873-9dedf8b9447a" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.118:54824 10.132.1.118:443 10.132.0.12:59060 - v1-completions-path [2026-11-02T08:50:31.061Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 65 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "1fdacd27-7145-4069-b975-34f491ea6e4f" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.118:54824 10.132.1.118:443 10.132.0.12:59070 - v1-completions-path 2026-11-02T08:50:43.332191Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:50:45.741875Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:51:10.411554Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:51:15.016329Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:51:21.126554Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:51:28.161816Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:51:33.612740Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:51:51.870764Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:51:54.922254Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:52:11.036494Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:52:14.122011Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:52:22.840737Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:52:28.191203Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:52:28.191249Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:52:28.191271Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T08:52:28.191279Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T08:52:28.112Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 310 78 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "ab54fb1d-4836-4e9b-b4b1-0e3110b5a5d1" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.118:56348 10.132.1.118:443 10.132.0.12:41312 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T08:52:40.660275Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:52:51.091691Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:53:07.633545Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:53:11.091830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:54:16.845311Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-02T08:55:31.941549Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:55:35.371906Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:56:39.939491Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T08:56:39.939521Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T08:56:39.939525Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T08:56:39.939529Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T08:56:39.894Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 403 - - "-" 73 113 44 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "d15eca43-7f90-4c38-b532-738c142b0ec3" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.118:443 10.132.0.12:48770 - v1-completions-path 2026-11-02T08:57:00.732965Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:57:16.211460Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:57:18.689725Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T08:57:18.721Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 119 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "453a9008-928a-4691-8093-273b40744ece" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.118:51460 10.132.1.118:443 10.132.0.12:41260 - v1-completions-path 2026-11-02T08:57:26.143694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:57:42.971122Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:57:46.114500Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:57:49.626343Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:57:49.626385Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:57:49.626400Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T08:57:49.626406Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T08:57:49.546Z] "DELETE /v1/api-keys/5a894e48-1a35-4b62-a3a7-0336437aea6d HTTP/1.1" 200 - via_upstream - "-" 0 369 80 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "b90acf47-f210-45d7-94e5-293672a7b39b" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.134.0.189:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.118:52924 10.132.1.118:443 10.132.0.12:50842 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T08:58:01.831230Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:58:12.494584Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:58:17.266014Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:58:34.429873Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:58:39.208035Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:58:42.846315Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:59:05.250389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:59:15.601207Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:59:30.228041Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:59:33.434771Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T08:59:38.623255Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T08:59:43.381Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 66 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "23ab08c1-3829-47b9-b25c-1cc81b442859" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.118:45880 10.132.1.118:443 10.132.0.12:46284 - v1-completions-path 2026-11-02T08:59:56.911941Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:00:07.288945Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:00:09.820012Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:00:22.232924Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:00:25.887988Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:00:44.350215Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:01:04.340167Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:01:14.694848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:01:25.149942Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:01:35.667909Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:01:46.132621Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:01:56.430133Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:02:06.823131Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:02:17.202041Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:02:27.487614Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T09:02:37.798131Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T09:02:47.106Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 53 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "3e3e6d57-e87e-44da-b853-3420be53ab94" "maas.apps.42d731be0d545102950e.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.118:443 10.132.0.12:55050 - v1-completions-path