2026-10-30T06:13:26.982315Z info FLAG: --concurrency="0" 2026-10-30T06:13:26.982337Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T06:13:26.982342Z info FLAG: --help="false" 2026-10-30T06:13:26.982346Z info FLAG: --log_as_json="false" 2026-10-30T06:13:26.982349Z info FLAG: --log_caller="" 2026-10-30T06:13:26.982352Z info FLAG: --log_output_level="default:info" 2026-10-30T06:13:26.982356Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T06:13:26.982364Z info FLAG: --log_target="[stdout]" 2026-10-30T06:13:26.982367Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T06:13:26.982370Z info FLAG: --outlierLogPath="" 2026-10-30T06:13:26.982374Z info FLAG: --profiling="true" 2026-10-30T06:13:26.982377Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T06:13:26.982380Z info FLAG: --proxyLogLevel="warning" 2026-10-30T06:13:26.982383Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T06:13:26.982386Z info FLAG: --stsPort="0" 2026-10-30T06:13:26.982389Z info FLAG: --templateFile="" 2026-10-30T06:13:26.982392Z info FLAG: --tokenManagerPlugin="" 2026-10-30T06:13:26.982396Z info FLAG: --vklog="0" 2026-10-30T06:13:26.982400Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T06:13:26.982580Z info Proxy role ips=[10.132.0.137] type=router id=maas-default-gateway-openshift-default-59f957fdd7-fp7hf.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T06:13:26.982643Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T06:13:26.984256Z info cpu limit detected as 2, setting concurrency 2026-10-30T06:13:26.984559Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T06:13:26.984572Z info JWT policy is third-party-jwt 2026-10-30T06:13:26.984576Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T06:13:26.987302Z info Starting default Istio SDS Server 2026-10-30T06:13:26.987337Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T06:13:26.987362Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T06:13:26.987973Z info Opening status port 15020 2026-10-30T06:13:26.988451Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T06:13:26.989311Z info sds Starting SDS grpc server 2026-10-30T06:13:26.989444Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T06:13:26.990344Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T06:13:26.991422Z info Starting proxy agent 2026-10-30T06:13:26.991511Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T06:13:27.057416Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T06:13:27.073081Z info cache generated new workload certificate resourceName=default latency=83.504184ms ttl=23h59m59.926923981s 2026-10-30T06:13:27.073218Z info cache Root cert has changed, start rotating root cert 2026-10-30T06:13:27.073268Z info cache returned workload trust anchor from cache ttl=23h59m59.926732787s 2026-10-30T06:13:27.306684Z info ads ADS: new connection for node:1 2026-10-30T06:13:27.331874Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T06:13:27.331971Z info cache read certificate from file resource=file-root:system 2026-10-30T06:13:27.407667Z info ads ADS: new connection for node:2 2026-10-30T06:13:27.407752Z info cache returned workload certificate from cache ttl=23h59m59.592250329s 2026-10-30T06:13:27.408297Z info ads ADS: new connection for node:3 2026-10-30T06:13:27.408383Z info cache returned workload trust anchor from cache ttl=23h59m59.591618261s 2026-10-30T06:13:27.524326Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:13:37.220612Z info Readiness succeeded in 10.241184793s 2026-10-30T06:13:37.221041Z info Envoy proxy is ready 2026-10-30T06:13:37.338604Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:14:09.221722Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:14:24.227612Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:15:07.640620Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:15:23.152714Z info ads ADS: new connection for node:4 2026-10-30T06:15:23.152924Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T06:15:23.152955Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T06:17:20.849650Z info ads ADS: "" 4 terminated with stream closed 2026-10-30T06:17:44.352296Z info ads ADS: "" 1 terminated with stream closed 2026-10-30T06:26:02.628299Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:26:41.114748Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:27:54.890428Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:35:18.361339Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:35:41.684900Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:36:13.062240Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:36:16.440Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 147 412 99 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "877ff7ce-e229-47d7-8fa9-c50fcc867505" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.133.0.41:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:43714 10.132.0.137:443 10.132.0.20:41564 - v1-chat-completions-path 2026-10-30T06:36:29.219849Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:36:31.794569Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:36:40.031909Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-10-30T06:36:39.947Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 91 3616 85 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "0f67cc85-a825-4bd3-80e9-9db633ec8749" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:35342 10.132.0.137:443 10.132.0.20:38254 - v1-catch-all-model-routing 2026-10-30T06:36:42.515677Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:37:00.519619Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:37:02.956454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:37:17.828155Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:37:28.807186Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:37:31.347851Z info ads ADS: new connection for node:5 2026-10-30T06:37:31.347978Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T06:37:35.279547Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:38:13.207122Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:38:32.353402Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:38:32.457594Z info ads ADS: "" 5 terminated 2026-10-30T06:38:35.804548Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:38:43.706203Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:40:01.014308Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:40:01.014342Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:40:01.014354Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T06:40:01.014359Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T06:40:00.949Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 64 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "6cec18f8-0f70-4c3a-82b6-e55b957adc1f" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.132.0.48:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.137:59668 10.132.0.137:443 10.132.0.20:47856 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:40:05.615564Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:40:23.727121Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:40:38.655397Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:40:50.648324Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:40:54.746612Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:41:06.447106Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:41:10.350494Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:41:17.743Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 73 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "6b54d5f1-9c00-46ff-a30b-421b0dedf94e" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:33274 10.132.0.137:443 10.132.0.20:36494 - v1-completions-path 2026-10-30T06:41:30.182420Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:42:16.013892Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T06:42:17.001869Z info ads ADS: new connection for node:6 2026-10-30T06:42:17.007086Z info cache read certificate from file resource=file-root:system 2026-10-30T06:42:18.030524Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:42:19.760467Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:42:24.788433Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"478","StreamId":"5994658558847296321"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-10-30T06:42:24.717Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 71 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "19d69f9b-e2de-4f8a-ad8a-a6e1cec9baaf" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.0.137:443 10.132.0.20:51940 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T06:42:31.087299Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-10-30T06:42:31.087530Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:42:31.087550Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:42:31.087560Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:42:31.087565Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T06:42:31.087706Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T06:42:30.983Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 104 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "3e303911-04b8-4ef1-9191-a98a160fbdb9" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.137:45952 10.132.0.137:443 10.132.0.20:51958 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T06:42:46.256095Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:42:46.256127Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:42:46.256136Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:42:46.256159Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T06:42:46.256306Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T06:42:46.139Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 116 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e679d7fc-f950-4c31-aba9-7f3f3ba438f0" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.137:59140 10.132.0.137:443 10.132.0.20:40600 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T06:42:52.628644Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-30T06:42:52.629116Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:42:52.629138Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:42:52.629217Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T06:42:52.629225Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:42:52.629378Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T06:42:52.538Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1728 91 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "80701422-9a97-4358-ba1c-5f3aa2e58183" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.137:59140 10.132.0.137:443 10.132.0.20:40628 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T06:43:01.124075Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:43:01.124109Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:43:01.124122Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:43:01.124128Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T06:43:01.124313Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T06:43:01.030Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 93 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "dde47d4c-f995-4080-9df3-851c486f35b8" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.137:45952 10.132.0.137:443 10.132.0.20:42776 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T06:43:35.121287Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:43:35.121318Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:43:35.121327Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T06:43:35.121331Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:43:35.121467Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T06:43:35.027Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1715 93 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "199360b7-ca4e-412c-8611-de75dbcb1e67" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.137:59140 10.132.0.137:443 10.132.0.20:49492 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T06:43:48.393203Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:43:48.515664Z info ads ADS: "" 6 terminated with stream closed 2026-10-30T06:43:50.121696Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:43:57.017701Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:44:14.090115Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:44:18.971165Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:44:39.054017Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:44:53.084793Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:45:00.066Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 62 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "c1fae4b5-1e06-4b13-abb7-2d184368d37b" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:33274 10.132.0.137:443 10.132.0.20:55154 - v1-completions-path [2026-10-30T06:45:01.066Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 298 58 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "4078f2dc-f2cf-4183-bdfa-1f005e5230fc" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:33274 10.132.0.137:443 10.132.0.20:55164 - v1-completions-path 2026-10-30T06:45:16.406118Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:45:26.831937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:45:31.653696Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:45:35.617Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 58 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "989b8799-b8b6-4908-9160-487c1ec2dbf0" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:33274 10.132.0.137:443 10.132.0.20:58716 - v1-completions-path [2026-10-30T06:45:36.095Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 70 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "48c30533-e4a4-4ce9-ac10-9ca84af0d899" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:38336 10.132.0.137:443 10.132.0.20:58738 - v1-completions-path 2026-10-30T06:45:48.120639Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:45:50.533889Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:46:09.674803Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:46:14.617087Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:46:28.033Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 299 79 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "188aa7b7-51d4-9c42-a50a-0899f13cea6a" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.133.0.42:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:60758 10.132.0.137:443 10.132.0.20:48892 - v1-completions-path 2026-10-30T06:46:30.546290Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:46:46.579823Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:46:58.152094Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:47:01.607530Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:47:09.666640Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:47:09.666676Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:47:09.666689Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:47:09.666694Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T06:47:09.586Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 324 80 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "7cf8eece-e509-4a4e-8a6d-451aa4e95c9f" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.132.0.48:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.137:59812 10.132.0.137:443 10.132.0.20:47616 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:47:22.567438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:47:25.531707Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:47:37.567848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:47:55.529320Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:48:05.777758Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:48:25.958938Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:48:29.343762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:49:59.667399Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:50:03.396260Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:51:33.168603Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:51:33.653Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 73 296 133 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "8bfce0cc-adbe-4744-97c3-b8b25743dbd7" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.133.0.40:8000" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:50406 10.132.0.137:443 10.132.0.20:38078 - v1-completions-path 2026-10-30T06:51:47.841734Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:51:50.310840Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:52:09.179753Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:52:21.362111Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:52:24.429581Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:52:27.703Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 56 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "1f86f5a2-fb41-4a73-8330-794b5e85b491" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.137:443 10.132.0.20:60678 - v1-completions-path 2026-10-30T06:52:40.053728Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:52:50.766500Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:52:55.486270Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:52:59.836Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 66 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "9769fd3e-e20f-47a7-a423-b707f1610837" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:33274 10.132.0.137:443 10.132.0.20:51294 - v1-completions-path [2026-10-30T06:53:00.333Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 96 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "2176704d-e1c5-4fd1-8259-6923a585949b" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:38336 10.132.0.137:443 10.132.0.20:51300 - v1-completions-path 2026-10-30T06:53:00.973846Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:53:00.973881Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:53:00.973896Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:53:00.973901Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T06:53:00.902Z] "DELETE /v1/api-keys/6920402f-0c91-4e8f-9181-e5cf0bbcee39 HTTP/1.1" 200 - via_upstream - "-" 0 365 70 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "88cbae46-1fb2-4ff4-a494-3709f785e6f3" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.132.0.169:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.137:40560 10.132.0.137:443 10.132.0.20:51314 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:53:13.000352Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:53:17.796461Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:53:21.371300Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:53:27.148Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 62 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "a5cee96b-b9e7-4366-a182-b587084954fe" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:33274 10.132.0.137:443 10.132.0.20:37070 - v1-completions-path [2026-10-30T06:53:31.372Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 62 - "52.71.36.72,10.132.0.20" "python-requests/2.34.2" "e4e06bc7-a3bc-425e-af31-b3aac085bdec" "maas.apps.32c8459a3857cfd60680.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.137:33274 10.132.0.137:443 10.132.0.20:37080 - v1-completions-path 2026-10-30T06:53:43.856701Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:53:54.102594Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:53:58.852631Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:54:02.529705Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:54:07.289045Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:54:25.694778Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:54:36.001103Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:54:38.386248Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:54:48.290768Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:54:52.683909Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:55:08.638774Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:55:11.157819Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:55:29.422991Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:55:39.656178Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:55:50.095021Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:56:00.333608Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:56:10.713726Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:56:20.979761Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:56:31.375518Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:56:41.692220Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:56:52.058888Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:57:02.547104Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8