2026-11-03T00:19:41.419430Z info FLAG: --concurrency="0" 2026-11-03T00:19:41.419452Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T00:19:41.419456Z info FLAG: --help="false" 2026-11-03T00:19:41.419459Z info FLAG: --log_as_json="false" 2026-11-03T00:19:41.419461Z info FLAG: --log_caller="" 2026-11-03T00:19:41.419463Z info FLAG: --log_output_level="default:info" 2026-11-03T00:19:41.419465Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T00:19:41.419469Z info FLAG: --log_target="[stdout]" 2026-11-03T00:19:41.419471Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T00:19:41.419473Z info FLAG: --outlierLogPath="" 2026-11-03T00:19:41.419475Z info FLAG: --profiling="true" 2026-11-03T00:19:41.419477Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T00:19:41.419508Z info FLAG: --proxyLogLevel="warning" 2026-11-03T00:19:41.419510Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T00:19:41.419512Z info FLAG: --stsPort="0" 2026-11-03T00:19:41.419514Z info FLAG: --templateFile="" 2026-11-03T00:19:41.419516Z info FLAG: --tokenManagerPlugin="" 2026-11-03T00:19:41.419518Z info FLAG: --vklog="0" 2026-11-03T00:19:41.419521Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T00:19:41.419669Z info Proxy role ips=[10.134.0.205] type=router id=maas-default-gateway-openshift-default-694db55c87-ksnhk.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T00:19:41.419724Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T00:19:41.420799Z info cpu limit detected as 2, setting concurrency 2026-11-03T00:19:41.421030Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T00:19:41.421041Z info JWT policy is third-party-jwt 2026-11-03T00:19:41.421044Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T00:19:41.422552Z info Starting default Istio SDS Server 2026-11-03T00:19:41.422580Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T00:19:41.422610Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T00:19:41.423287Z info Opening status port 15020 2026-11-03T00:19:41.424108Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T00:19:41.425398Z info sds Starting SDS grpc server 2026-11-03T00:19:41.425419Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T00:19:41.426132Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T00:19:41.427258Z info Starting proxy agent 2026-11-03T00:19:41.427298Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T00:19:41.493249Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T00:19:41.544879Z info cache generated new workload certificate resourceName=default latency=119.281921ms ttl=23h59m59.455126554s 2026-11-03T00:19:41.544917Z info cache Root cert has changed, start rotating root cert 2026-11-03T00:19:41.544953Z info cache returned workload trust anchor from cache ttl=23h59m59.455047254s 2026-11-03T00:19:41.678693Z info ads ADS: new connection for node:1 2026-11-03T00:19:41.678765Z info cache returned workload certificate from cache ttl=23h59m59.321237436s 2026-11-03T00:19:41.679234Z info ads ADS: new connection for node:2 2026-11-03T00:19:41.679309Z info cache returned workload trust anchor from cache ttl=23h59m59.320692114s 2026-11-03T00:19:41.762137Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:19:49.797464Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:19:50.919889Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-03T00:19:51.742426Z info Readiness succeeded in 10.325442158s 2026-11-03T00:19:51.742849Z info Envoy proxy is ready 2026-11-03T00:19:53.702984Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:19:53.703024Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:19:53.703035Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:19:53.703042Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T00:19:53.574Z] "POST /v1/api-keys HTTP/1.1" 400 - via_upstream - "-" 80 94 128 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "8ef23c63-53f4-4b4a-a09b-2be087ee3658" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.134.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.205:60126 10.134.0.205:443 10.132.0.17:49468 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:20:20.570350Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:20:23.715422Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:20:23.715465Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:20:23.715498Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T00:20:23.715503Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T00:20:23.577Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 61 304 138 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "50016aa8-62d9-4705-a067-bf1ab0441bf5" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.134.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.205:33620 10.134.0.205:443 10.132.0.17:38172 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T00:20:24.213Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 2817 3089 77 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "10cb7e91-3487-45d7-a76c-333ec4930f96" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.132.0.62:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:41208 10.134.0.205:443 10.132.0.17:38180 - v1-chat-completions-path [2026-11-03T00:20:25.529Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 108118 108394 329 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "c2101297-dc71-4f46-93da-a45a1a939e1e" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.132.0.62:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:41208 10.134.0.205:443 10.132.0.17:38190 - v1-chat-completions-path 2026-11-03T00:20:38.790150Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T00:20:38.790183Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T00:20:38.790187Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T00:20:38.790192Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T00:20:38.639Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 151 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "cd54bfd2-9a24-4259-abe0-bdb804e182c8" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.205:443 10.132.0.17:60272 - v1-catch-all-path 2026-11-03T00:20:40.349281Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:20:45.302197Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:20:54.824524Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:20:57.965930Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T00:20:57.841Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 124 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "97bf5b92-69f3-4419-b4ed-797945241010" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:36614 10.134.0.205:443 10.132.0.17:33944 - v1-catch-all-path 2026-11-03T00:20:59.138378Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T00:20:59.143270Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T00:20:59.066Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 76 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "4e217702-6049-48a2-82cf-2c591a9e2f20" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:36630 10.134.0.205:443 10.132.0.17:39820 - v1-catch-all-path 2026-11-03T00:21:08.694946Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:21:18.125774Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:21:28.683166Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:21:46.648856Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:21:48.590212Z info ads ADS: new connection for node:3 2026-11-03T00:21:48.590407Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T00:21:48.590441Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T00:21:56.164748Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:22:32.559960Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:22:56.484134Z info ads ADS: "" 3 terminated with stream closed 2026-11-03T00:22:56.545745Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:23:01.375247Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:23:17.865820Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:25:57.201630Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:26:28.512601Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:26:56.975741Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:26:56.975787Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:26:56.975805Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T00:26:56.975810Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T00:26:56.908Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 628 66 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "9e8fc68d-12ee-4ad9-8ba7-4c7b4eec53a2" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.134.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.205:35762 10.134.0.205:443 10.132.0.17:54942 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:26:59.127002Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:27:14.581733Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:27:31.562079Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:27:44.099662Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:27:49.000195Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:27:53.975390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-03T00:27:53.900Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 76 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "eeb320d1-2f7c-418e-97e1-fa46f1399a7b" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:46280 10.134.0.205:443 10.132.0.17:34232 - v1-completions-path 2026-11-03T00:28:09.482920Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:28:26.526Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 79 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "cee67e7c-50e2-4417-ba78-428a6742e159" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.132.0.63:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:37334 10.134.0.205:443 10.132.0.17:44308 - v1-completions-path [2026-11-03T00:28:40.586Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 287 74 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "bdf85ec4-bb37-472d-bfbf-2f139b6cf6d6" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.132.0.63:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:37334 10.134.0.205:443 10.132.0.17:58336 - v1-completions-path 2026-11-03T00:28:56.855068Z info ads ADS: new connection for node:4 2026-11-03T00:28:56.860664Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T00:28:56.860738Z info cache read certificate from file resource=file-root:system 2026-11-03T00:29:00.775398Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:01.994721Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:05.024383Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"184","StreamId":"6187811405225847221"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-03T00:29:04.961Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 63 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "8f168621-fd8d-43f0-b98c-9dd0ed92cdf6" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.134.0.205:443 10.132.0.17:60852 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T00:29:39.633301Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:29:39.633343Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:29:39.633359Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:29:39.633365Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T00:29:39.633611Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T00:29:39.508Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1723 125 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "1ad0d450-f1b1-4345-ac3a-6b95eb42ba01" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.205:59140 10.134.0.205:443 10.132.0.17:36046 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T00:29:55.998292Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:29:55.998330Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:29:55.998340Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:29:55.998344Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T00:29:55.998524Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T00:29:55.887Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1721 111 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "91a17cc6-f833-42f0-9783-58100018e904" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.205:59140 10.134.0.205:443 10.132.0.17:41318 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T00:30:57.494323Z info ads ADS: "" 4 terminated 2026-11-03T00:30:57.545633Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:31:00.251241Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:31:10.913770Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:31:29.408727Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:31:34.036575Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:31:38.398Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 84 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "dee95633-9074-401b-b68d-77a9da8d3b12" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:38370 10.134.0.205:443 10.132.0.17:43682 - v1-completions-path [2026-11-03T00:31:38.685Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 289 64 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "18b8ceb8-4b21-4493-87a6-ec22024ad88b" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:46280 10.134.0.205:443 10.132.0.17:43690 - v1-completions-path [2026-11-03T00:31:39.730Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 58 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "c6f1513b-d1a5-4252-9b4b-d4e7f04033ac" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:38370 10.134.0.205:443 10.132.0.17:56896 - v1-completions-path 2026-11-03T00:31:58.146944Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:32:11.342070Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:32:23.656Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 64 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "e7d9b1a9-76e4-4181-a666-476208186aa8" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:38370 10.134.0.205:443 10.132.0.17:57034 - v1-completions-path 2026-11-03T00:32:42.211129Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:32:52.921574Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:32:59.427843Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:33:27.767419Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:33:32.330905Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:33:45.659101Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:33:52.259358Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:06.712968Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:11.435302Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:24.971566Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:38.368296Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:44.327761Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:02.776961Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:35:05.062Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 66 18 58 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "986dfdc6-8896-4ee5-bad9-e7432b9d9cbb" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.205:443 10.132.0.17:41248 - v1-completions-path 2026-11-03T00:35:08.403186Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:19.854577Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:46.385676Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:57.047380Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:36:15.712458Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:36:23.428580Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:39:20.102879Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:40:26.954151Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T00:40:26.954176Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T00:40:26.954179Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T00:40:26.954183Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T00:40:26.908Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 403 - - "-" 73 113 45 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "f2b0eb54-83d4-4443-acd8-e545c004effe" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.205:443 10.132.0.17:59152 - v1-completions-path 2026-11-03T00:40:48.538259Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:41:05.900794Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:41:10.681460Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:41:11.365Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 295 96 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "9520c47d-e4a9-4b4f-813d-6c96eb7f4900" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.132.0.63:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:37850 10.134.0.205:443 10.132.0.17:60814 - v1-completions-path 2026-11-03T00:41:24.031633Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:41:39.854164Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:41:44.744743Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:41:47.801501Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:41:47.801542Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:41:47.801557Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T00:41:47.801562Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T00:41:47.676Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 314 125 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "7ab5cbca-4c22-43a2-bea6-953a0912158c" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.132.0.243:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.205:39226 10.134.0.205:443 10.132.0.17:56614 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:42:04.667614Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:42:15.470382Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:42:21.792855Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:42:50.991253Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:42:57.423565Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:43:03.538677Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:43:12.256Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 64 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "8458a42a-e81b-441c-8f6c-93e757d2ea08" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:46280 10.134.0.205:443 10.132.0.17:39184 - v1-completions-path [2026-11-03T00:43:14.833Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 64 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "83aed680-a3cb-4311-9c97-86ecd6e73180" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.205:38370 10.134.0.205:443 10.132.0.17:39188 - v1-completions-path [2026-11-03T00:43:21.367Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 63 - "44.212.242.249,10.132.0.17" "python-requests/2.34.2" "0b1f20d1-7333-4798-af43-ecd3abc9db37" "maas.apps.0f3a63f01c0b122488b1.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.205:443 10.132.0.17:55664 - v1-completions-path 2026-11-03T00:43:35.694133Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:43:46.271721Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:43:52.806770Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:43:58.755195Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:44:11.047395Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:44:34.487369Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:44:45.111754Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:44:49.990275Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:45:08.361537Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:45:15.355006Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:45:24.770839Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:45:45.096596Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:45:57.798696Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:46:17.929066Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:46:28.971801Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:46:39.907128Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:46:50.587572Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:47:01.371218Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:47:12.151822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:47:22.903629Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:47:33.536354Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8