2026-11-04T01:32:33.017263Z info FLAG: --concurrency="0" 2026-11-04T01:32:33.017293Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-04T01:32:33.017297Z info FLAG: --help="false" 2026-11-04T01:32:33.017299Z info FLAG: --log_as_json="false" 2026-11-04T01:32:33.017302Z info FLAG: --log_caller="" 2026-11-04T01:32:33.017304Z info FLAG: --log_output_level="default:info" 2026-11-04T01:32:33.017306Z info FLAG: --log_stacktrace_level="default:none" 2026-11-04T01:32:33.017310Z info FLAG: --log_target="[stdout]" 2026-11-04T01:32:33.017312Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-04T01:32:33.017314Z info FLAG: --outlierLogPath="" 2026-11-04T01:32:33.017316Z info FLAG: --profiling="true" 2026-11-04T01:32:33.017319Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-04T01:32:33.017320Z info FLAG: --proxyLogLevel="warning" 2026-11-04T01:32:33.017322Z info FLAG: --serviceCluster="istio-proxy" 2026-11-04T01:32:33.017324Z info FLAG: --stsPort="0" 2026-11-04T01:32:33.017326Z info FLAG: --templateFile="" 2026-11-04T01:32:33.017328Z info FLAG: --tokenManagerPlugin="" 2026-11-04T01:32:33.017331Z info FLAG: --vklog="0" 2026-11-04T01:32:33.017333Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-04T01:32:33.017476Z info Proxy role ips=[10.133.0.185] type=router id=maas-default-gateway-openshift-default-5d84bc6b95-djmbj.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-04T01:32:33.017522Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-04T01:32:33.018909Z info cpu limit detected as 2, setting concurrency 2026-11-04T01:32:33.019095Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-04T01:32:33.019121Z info JWT policy is third-party-jwt 2026-11-04T01:32:33.019124Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-04T01:32:33.020665Z info Starting default Istio SDS Server 2026-11-04T01:32:33.020685Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-04T01:32:33.020708Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-04T01:32:33.021066Z info Opening status port 15020 2026-11-04T01:32:33.022002Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-04T01:32:33.023469Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-04T01:32:33.024373Z info Starting proxy agent 2026-11-04T01:32:33.024403Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-04T01:32:33.033479Z info sds Starting SDS grpc server 2026-11-04T01:32:33.033496Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-04T01:32:33.087652Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-04T01:32:33.149779Z info cache generated new workload certificate resourceName=default latency=127.976308ms ttl=23h59m59.850226281s 2026-11-04T01:32:33.149814Z info cache Root cert has changed, start rotating root cert 2026-11-04T01:32:33.149846Z info cache returned workload trust anchor from cache ttl=23h59m59.850154454s 2026-11-04T01:32:33.257310Z info ads ADS: new connection for node:1 2026-11-04T01:32:33.257387Z info cache returned workload certificate from cache ttl=23h59m59.742614649s 2026-11-04T01:32:33.257723Z info ads ADS: new connection for node:2 2026-11-04T01:32:33.257803Z info cache returned workload trust anchor from cache ttl=23h59m59.74219877s 2026-11-04T01:32:33.337526Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:32:41.143897Z info Readiness succeeded in 8.128932365s 2026-11-04T01:32:41.144243Z info Envoy proxy is ready 2026-11-04T01:33:08.887720Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-04T01:33:13.688Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 144 409 90 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "8dc406b5-04ef-412c-b54b-ee274848e456" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.41:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:51300 10.133.0.185:443 10.132.0.9:48380 - v1-chat-completions-path 2026-11-04T01:33:26.391367Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-04T01:33:26.391396Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-04T01:33:26.391400Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-04T01:33:26.391404Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-04T01:33:26.238Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 153 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "5a743b96-56bd-4314-997c-d4a762fddfc1" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.185:443 10.132.0.9:46340 - v1-catch-all-path 2026-11-04T01:33:27.809007Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:33:32.674477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:33:35.195972Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 2026-11-04T01:33:35.668969Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-04T01:33:35.085Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 110 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "cb00cb69-47bf-439e-bfa1-4e4edb9f25f1" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:54328 10.133.0.185:443 10.132.0.9:59874 - v1-catch-all-path [2026-11-04T01:33:35.569Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 99 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "a44e72c4-c691-40ae-bd4a-42fc0d3be4f2" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:54328 10.133.0.185:443 10.132.0.9:59878 - v1-catch-all-path 2026-11-04T01:33:42.619830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:33:44.714083Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-04T01:33:44.714395Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-04T01:33:44.632Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 82 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "3e45ed1e-a6a5-9f79-9449-ec345b0b84f2" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:54328 10.133.0.185:443 10.132.0.9:59544 - v1-catch-all-path 2026-11-04T01:33:45.122965Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-04T01:33:45.032Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 90 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "565ef365-579a-498d-bee6-b3468ad43982" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:54328 10.133.0.185:443 10.132.0.9:59560 - v1-catch-all-path 2026-11-04T01:33:45.976189Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-04T01:33:45.892Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 83 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "ea35e645-8e54-4370-b2f4-4d68939162eb" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:46236 10.133.0.185:443 10.132.0.9:59566 - v1-catch-all-path [2026-11-04T01:33:48.590Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 429 - - "-" 61 18 70 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "1214f434-25d5-4768-b8cc-289cdef11c90" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.185:443 10.132.0.9:59572 - v1-catch-all-path 2026-11-04T01:33:57.129563Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:34:06.910511Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:34:09.444162Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-04T01:34:09.244Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 200 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "dd38d3e2-1335-4926-941c-5355e1082964" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:46236 10.133.0.185:443 10.132.0.9:60840 - v1-catch-all-path 2026-11-04T01:34:23.163962Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:34:35.095925Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:34:37.043958Z info ads ADS: new connection for node:3 2026-11-04T01:34:37.044631Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-04T01:34:37.044759Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-04T01:34:40.497740Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:35:21.324418Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-04T01:35:31.619Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 347 85 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "d691fb22-c9e1-4acf-8274-8b58ae221a1c" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.133.0.188:8000" outbound|54321||e2e-prio-pool-9e0c44d6-inference-pool-ip-b76b9531.llm.svc.cluster.local 10.133.0.185:39904 10.133.0.185:443 10.132.0.9:40822 - v1-model-routing [2026-11-04T01:35:43.586Z] "POST /v1/chat/completions HTTP/1.1" 403 - - "-" 131 0 36 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "568b86e6-6ab7-41d8-b049-096d35514b58" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "-" outbound|54321||e2e-prio-pool-9e0c44d6-inference-pool-ip-b76b9531.llm.svc.cluster.local - 10.133.0.185:443 10.132.0.9:41048 - v1-model-routing 2026-11-04T01:35:45.744078Z info ads ADS: "" 3 terminated 2026-11-04T01:35:46.146375Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-04T01:35:50.971Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 75 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "e1589abb-a604-4785-a4a0-c1b8161c8e17" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.132.0.51:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:50636 10.133.0.185:443 10.132.0.9:48946 - v1-completions-path 2026-11-04T01:35:51.441056Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:36:08.181630Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:38:42.461691Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:39:10.816682Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:39:30.469984Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:39:44.465982Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:39:51.528905Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:40:02.459702Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:40:07.539143Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-04T01:40:11.155Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 72 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "2f6e05f1-d8c6-4c63-874d-640bcdfe0a49" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:46108 10.133.0.185:443 10.132.0.9:41102 - v1-completions-path 2026-11-04T01:40:25.674066Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:41:13.591640Z info ads ADS: new connection for node:4 2026-11-04T01:41:13.595988Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-04T01:41:13.596041Z info cache read certificate from file resource=file-root:system 2026-11-04T01:41:17.278480Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:41:18.876797Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:41:28.695651Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"177","StreamId":"11725683523177140946"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-04T01:41:28.633Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 62 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "432a30aa-ab0c-41cc-895b-44bf64acb16e" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.185:443 10.132.0.9:34960 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-04T01:41:31.084147Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:41:31.084181Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:41:31.084193Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:41:31.084198Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T01:41:31.084381Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-04T01:41:30.885Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1535 199 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "f2882441-0e4a-491a-bc04-2579208f4f3c" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.185:60776 10.133.0.185:443 10.132.0.9:44604 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-04T01:41:50.345025Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:41:50.345058Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:41:50.345069Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:41:50.345073Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T01:41:50.345244Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-04T01:41:50.221Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1529 123 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "62cfadc4-b33b-44c6-a5f8-209a35773b7a" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.185:60776 10.133.0.185:443 10.132.0.9:34756 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-04T01:42:07.078744Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-04T01:42:07.206230Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:42:07.206262Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:42:07.206272Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:42:07.206277Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:42:07.206448Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-04T01:42:07.019Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1535 186 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "b307650f-0315-40f5-878a-414fc6f56b4d" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.185:48908 10.133.0.185:443 10.132.0.9:56556 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-04T01:42:55.585349Z info ads ADS: "" 4 terminated with stream closed 2026-11-04T01:42:57.103077Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:42:58.370681Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:43:09.283647Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:43:27.777330Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:43:32.700191Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:43:57.280272Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:44:10.437700Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:44:41.601224Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:44:52.303013Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:44:57.390983Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:45:09.178982Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:45:09.179017Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:45:09.179029Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:45:09.179034Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-04T01:45:09.063Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 318 115 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "76af00a0-0d8b-4bc5-9af8-bf86e5f9d743" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.133.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.185:54970 10.133.0.185:443 10.132.0.9:42432 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:45:11.221Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 61 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "c73150a9-d7b0-45e1-8f57-500e31d2fdf0" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:42626 10.133.0.185:443 10.132.0.9:58066 - v1-completions-path 2026-11-04T01:45:26.960575Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:45:31.943875Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:45:46.041097Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:45:55.466162Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:46:03.584859Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:46:12.925058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:46:22.656244Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:46:41.206440Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:46:43.827196Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:46:43.827229Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:46:43.827242Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:46:43.827246Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T01:46:43.696Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 324 130 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "a27dd916-8178-435a-bdca-4d88a7a7e029" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.133.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.185:46380 10.133.0.185:443 10.132.0.9:37240 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:47:12.729775Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:47:36.136116Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:47:46.843337Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:47:50.789144Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:47:50.789175Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:47:50.789187Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:47:50.789192Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-04T01:47:50.670Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 118 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "38808cec-32aa-416c-bd43-b7cbbc267cf3" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.133.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.185:34976 10.133.0.185:443 10.132.0.9:49428 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:52:47.804453Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:53:05.046263Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:53:09.615843Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:53:22.323207Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:53:39.991278Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:53:46.492354Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-04T01:53:51.024Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 60 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "c5a787e3-1b1f-4e22-b603-b94dcdd8cccb" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:42626 10.133.0.185:443 10.132.0.9:49092 - v1-completions-path 2026-11-04T01:53:52.344608Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:53:52.344652Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:53:52.344676Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T01:53:52.344683Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T01:53:52.232Z] "DELETE /v1/api-keys/d62371ba-b5d9-4a7e-a3bc-0ab16abd681e HTTP/1.1" 200 - via_upstream - "-" 0 369 112 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "638e479f-d385-46f3-9fe8-2b8048e98f07" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.133.0.192:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.185:49838 10.133.0.185:443 10.132.0.9:49108 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:54:05.967912Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:54:16.628882Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:54:22.845902Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:54:33.712647Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:54:33.712685Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:54:33.712700Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T01:54:33.712705Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T01:54:33.600Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 309 111 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "abf24918-ff5a-41ac-85f0-cfa87da4926e" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.133.0.192:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.185:49838 10.133.0.185:443 10.132.0.9:45118 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:54:50.763461Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:54:56.919313Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:55:02.787915Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-04T01:55:16.563Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 72 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "7abda21e-9bb8-42ef-9dfd-256ea528ac6d" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:46108 10.133.0.185:443 10.132.0.9:35124 - v1-completions-path 2026-11-04T01:55:34.481638Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:55:45.274185Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:55:51.713748Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:55:59.254495Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-04T01:56:06.693Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 68 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "41b463eb-d32b-4572-8f20-457156fa216c" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:42626 10.133.0.185:443 10.132.0.9:60146 - v1-completions-path 2026-11-04T01:56:10.446830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-04T01:56:17.261Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 64 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "ba9675ae-163d-49f5-99fd-e4303d5aa846" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.185:42626 10.133.0.185:443 10.132.0.9:56634 - v1-completions-path [2026-11-04T01:56:19.830Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 53 - "44.212.242.249,10.132.0.9" "python-requests/2.34.2" "53093f24-fc21-4784-9781-7e7908c96af8" "maas.apps.34861718878ef819d16d.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.185:443 10.132.0.9:56654 - v1-completions-path 2026-11-04T01:56:33.837305Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:56:44.624719Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:56:49.412165Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:57:07.824367Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:57:13.890146Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:57:35.975682Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:57:57.828491Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:58:18.135771Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:58:30.654012Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:58:50.671958Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:59:01.580588Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:59:12.292049Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:59:23.152549Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-04T01:59:33.754094Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7