2026-11-03T15:08:12.773135Z info FLAG: --concurrency="0" 2026-11-03T15:08:12.773164Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T15:08:12.773168Z info FLAG: --help="false" 2026-11-03T15:08:12.773171Z info FLAG: --log_as_json="false" 2026-11-03T15:08:12.773173Z info FLAG: --log_caller="" 2026-11-03T15:08:12.773175Z info FLAG: --log_output_level="default:info" 2026-11-03T15:08:12.773177Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T15:08:12.773181Z info FLAG: --log_target="[stdout]" 2026-11-03T15:08:12.773184Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T15:08:12.773185Z info FLAG: --outlierLogPath="" 2026-11-03T15:08:12.773188Z info FLAG: --profiling="true" 2026-11-03T15:08:12.773189Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T15:08:12.773191Z info FLAG: --proxyLogLevel="warning" 2026-11-03T15:08:12.773194Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T15:08:12.773196Z info FLAG: --stsPort="0" 2026-11-03T15:08:12.773198Z info FLAG: --templateFile="" 2026-11-03T15:08:12.773200Z info FLAG: --tokenManagerPlugin="" 2026-11-03T15:08:12.773202Z info FLAG: --vklog="0" 2026-11-03T15:08:12.773205Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T15:08:12.773376Z info Proxy role ips=[10.132.1.146] type=router id=maas-default-gateway-openshift-default-855f9574cf-lclzm.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T15:08:12.773424Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T15:08:12.774575Z info cpu limit detected as 2, setting concurrency 2026-11-03T15:08:12.775152Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T15:08:12.775164Z info JWT policy is third-party-jwt 2026-11-03T15:08:12.775167Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T15:08:12.777018Z info Starting default Istio SDS Server 2026-11-03T15:08:12.777049Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T15:08:12.777067Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T15:08:12.777694Z info Opening status port 15020 2026-11-03T15:08:12.778084Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T15:08:12.778845Z info sds Starting SDS grpc server 2026-11-03T15:08:12.778863Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T15:08:12.779649Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T15:08:12.780545Z info Starting proxy agent 2026-11-03T15:08:12.780578Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T15:08:12.846064Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T15:08:12.942088Z info cache generated new workload certificate resourceName=default latency=163.05315ms ttl=23h59m59.057917937s 2026-11-03T15:08:12.942128Z info cache Root cert has changed, start rotating root cert 2026-11-03T15:08:12.942162Z info cache returned workload trust anchor from cache ttl=23h59m59.057838056s 2026-11-03T15:08:12.995302Z info ads ADS: new connection for node:1 2026-11-03T15:08:12.995381Z info cache returned workload certificate from cache ttl=23h59m59.004620932s 2026-11-03T15:08:12.995826Z info ads ADS: new connection for node:2 2026-11-03T15:08:12.995902Z info cache returned workload trust anchor from cache ttl=23h59m59.004099302s 2026-11-03T15:08:13.095385Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:08:18.039308Z error failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused 2026-11-03T15:08:21.450883Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:08:22.373504Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-03T15:08:23.243629Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-03T15:08:23.637636Z info Readiness succeeded in 10.867411271s 2026-11-03T15:08:23.637996Z info Envoy proxy is ready 2026-11-03T15:08:32.407799Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T15:08:32.407829Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T15:08:32.407832Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T15:08:32.407836Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-03T15:08:32.250Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 156 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "243512ce-e060-4b1a-8b4c-8f5c47dc4032" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.146:443 10.132.0.14:53302 - v1-catch-all-path 2026-11-03T15:08:33.876016Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:08:38.737586Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:08:41.292519Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T15:08:41.202Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 90 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "53f10646-1d64-4cd2-a1f7-f8bee05310aa" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:49364 10.132.1.146:443 10.132.0.14:35630 - v1-catch-all-path 2026-11-03T15:08:48.336735Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:08:50.815639Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T15:08:50.729Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 86 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "c927cd10-cf4e-4709-8085-848bb6e1cd3c" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:49364 10.132.1.146:443 10.132.0.14:45516 - v1-catch-all-path 2026-11-03T15:08:51.567241Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 2026-11-03T15:08:52.334234Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-03T15:08:52.335553Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T15:08:51.497Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 69 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "1041291b-da91-4fff-8928-8edee4bcf0f2" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:58404 10.132.1.146:443 10.132.0.14:45532 - v1-catch-all-path [2026-11-03T15:08:52.269Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 65 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "1fa70350-ee77-46a9-aab3-462fd80eb249" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:49364 10.132.1.146:443 10.132.0.14:45548 - v1-catch-all-path 2026-11-03T15:08:53.791237Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T15:08:53.731Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 59 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "8c106c9d-81c0-49f4-bd25-0bd70afd4084" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:49364 10.132.1.146:443 10.132.0.14:45556 - v1-catch-all-path [2026-11-03T15:08:54.097Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 429 - - "-" 61 18 67 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "1d67b975-f76f-4961-a534-8cdc8bdcbda6" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.146:443 10.132.0.14:51304 - v1-catch-all-path 2026-11-03T15:09:01.805622Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:09:11.067272Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:09:13.711047Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-03T15:09:13.588Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 122 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "b3cfef49-dc7e-449e-89cf-01defebe6cb3" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:49364 10.132.1.146:443 10.132.0.14:52584 - v1-catch-all-path 2026-11-03T15:09:27.571269Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:09:39.223057Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:09:42.783469Z info ads ADS: new connection for node:3 2026-11-03T15:09:42.783707Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T15:09:42.783801Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T15:09:46.301617Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:10:30.613327Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:10:33.923394Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T15:10:33.923431Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T15:10:33.923448Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T15:10:33.923453Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T15:10:33.800Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 313 123 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "6b48d711-a435-44d9-86ae-87928ebf81fe" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.36:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.146:49894 10.132.1.146:443 10.132.0.14:51412 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T15:10:36.025Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 343 86 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "8c0479b8-71a9-4229-a894-2e19b66eadfd" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.132.1.153:8000" outbound|54321||e2e-prio-pool-3b12e11b-inference-pool-ip-f5e5c752.llm.svc.cluster.local 10.132.1.146:51972 10.132.1.146:443 10.132.0.14:36254 - v1-model-routing 2026-11-03T15:10:55.394239Z info ads ADS: "" 3 terminated with stream closed 2026-11-03T15:10:55.443477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:11:00.396563Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:11:14.354272Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:13:52.590250Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:14:23.181141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:14:54.403411Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:15:09.881415Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:15:21.990185Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:15:36.577844Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T15:15:43.205Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 79 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "0995dbe4-52dd-407e-92da-98c5667c78e7" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:38190 10.132.1.146:443 10.132.0.14:55834 - v1-completions-path 2026-11-03T15:15:43.774252Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:16:01.818999Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:16:50.596254Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-03T15:16:50.521Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 94 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "51c7ab6d-d514-9760-a60f-3127dbf00b1b" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:42138 10.132.1.146:443 10.132.0.14:38974 - v1-completions-path 2026-11-03T15:16:52.332170Z info ads ADS: new connection for node:4 2026-11-03T15:16:52.337530Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T15:16:52.337595Z info cache read certificate from file resource=file-root:system 2026-11-03T15:16:56.797863Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:16:58.037306Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:17:05.968314Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"186","StreamId":"10503925119142594454"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-03T15:17:05.906Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 62 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "627ef391-5fcf-4e21-ba4f-13d9cb1bffed" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.1.146:443 10.132.0.14:37348 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T15:17:17.557768Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T15:17:17.557800Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T15:17:17.557810Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T15:17:17.557814Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T15:17:17.558016Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T15:17:17.428Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1710 129 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "bd1b9e41-dd4e-48dd-b63a-63c598c6ad59" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.146:41780 10.132.1.146:443 10.132.0.14:33426 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T15:17:31.497770Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T15:17:31.497806Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T15:17:31.497817Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T15:17:31.497821Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T15:17:31.498020Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T15:17:31.382Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 115 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "e235a914-8d58-4a5c-abd1-a788316506e4" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.146:47022 10.132.1.146:443 10.132.0.14:32830 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T15:18:35.619856Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T15:18:35.619887Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T15:18:35.619896Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T15:18:35.619900Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T15:18:35.620106Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T15:18:35.497Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1707 122 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "f2a27fb8-cac5-4a82-8fd6-a5def0f7ba93" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.146:41780 10.132.1.146:443 10.132.0.14:55288 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T15:18:52.683187Z info ads ADS: "" 4 terminated 2026-11-03T15:18:52.726350Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:18:57.079567Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:19:05.711139Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:19:21.540417Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:19:26.273540Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:19:46.274765Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:20:03.326059Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T15:20:12.681Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "f46004c5-5378-47e8-90a8-9f82372a57c6" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:38190 10.132.1.146:443 10.132.0.14:52650 - v1-completions-path 2026-11-03T15:20:34.457874Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:20:45.299012Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:20:51.911449Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:21:23.225206Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:21:28.118025Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:21:41.790827Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T15:21:46.355Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 74 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "0d8d4047-33f0-4169-9a53-a4a59310f737" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:59420 10.132.1.146:443 10.132.0.14:50720 - v1-completions-path 2026-11-03T15:21:50.872823Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:21:58.953624Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:22:08.132463Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:22:16.233516Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:22:36.971304Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:23:08.640468Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:23:31.892694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:23:42.588294Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:28:35.650738Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:28:52.967118Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:28:59.438845Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T15:29:02.637Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 63 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "85e327cb-fac9-4e6f-968c-26aa8ac56c28" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:42138 10.132.1.146:443 10.132.0.14:51240 - v1-completions-path 2026-11-03T15:29:14.156282Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:29:29.572276Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:29:34.736064Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:29:54.681271Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:30:05.450135Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:30:11.831010Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:30:42.047792Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:30:48.480369Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:30:54.617844Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:31:09.758815Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T15:31:09.758854Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T15:31:09.758870Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T15:31:09.758875Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T15:31:09.640Z] "DELETE /v1/api-keys/51faa58e-a38f-4516-ab59-7b4e1cd1126e HTTP/1.1" 200 - via_upstream - "-" 0 363 118 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "f23f16cd-0c3a-4103-9887-f2249d6c2fd0" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.199:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.146:49396 10.132.1.146:443 10.132.0.14:55556 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T15:31:23.693494Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:31:34.361570Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:31:39.503822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:31:45.983596Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:31:54.297487Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T15:32:00.804Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 67 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "713ccdfe-e03f-48a5-9f96-6063d4933216" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:54186 10.132.1.146:443 10.132.0.14:44864 - v1-completions-path 2026-11-03T15:32:17.437912Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:32:28.383630Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:32:33.352104Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:32:51.653460Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:32:58.079942Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:33:19.560725Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:33:39.448009Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:33:50.459236Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:34:01.365540Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:34:12.162021Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:34:23.325886Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:34:34.034970Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:34:46.139986Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:34:56.045445Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:35:06.491200Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:35:17.184203Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T15:35:25.213274Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T15:35:25.213302Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T15:35:25.213305Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-03T15:35:25.213308Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-03T15:35:25.186Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 26 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "21a359f7-4562-4c46-8fab-3c56bd0291ea" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.146:443 10.132.0.14:34434 - v1-completions-path [2026-11-03T15:35:28.870Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 74 - "52.71.36.72,10.132.0.14" "python-requests/2.34.2" "37f975d0-9019-4dbe-b164-5fcf75439c67" "maas.apps.faf79739dd66677c5df2.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.146:42138 10.132.1.146:443 10.132.0.14:34442 - v1-completions-path