2026-11-02T23:49:14.164019Z info FLAG: --concurrency="0" 2026-11-02T23:49:14.164035Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T23:49:14.164040Z info FLAG: --help="false" 2026-11-02T23:49:14.164042Z info FLAG: --log_as_json="false" 2026-11-02T23:49:14.164044Z info FLAG: --log_caller="" 2026-11-02T23:49:14.164046Z info FLAG: --log_output_level="default:info" 2026-11-02T23:49:14.164048Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T23:49:14.164052Z info FLAG: --log_target="[stdout]" 2026-11-02T23:49:14.164055Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T23:49:14.164057Z info FLAG: --outlierLogPath="" 2026-11-02T23:49:14.164059Z info FLAG: --profiling="true" 2026-11-02T23:49:14.164061Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T23:49:14.164062Z info FLAG: --proxyLogLevel="warning" 2026-11-02T23:49:14.164064Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T23:49:14.164066Z info FLAG: --stsPort="0" 2026-11-02T23:49:14.164073Z info FLAG: --templateFile="" 2026-11-02T23:49:14.164075Z info FLAG: --tokenManagerPlugin="" 2026-11-02T23:49:14.164081Z info FLAG: --vklog="0" 2026-11-02T23:49:14.164084Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T23:49:14.164228Z info Proxy role ips=[10.133.0.120] type=router id=maas-default-gateway-openshift-default-55f85cf5d7-w8nxw.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T23:49:14.164273Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T23:49:14.165734Z info cpu limit detected as 2, setting concurrency 2026-11-02T23:49:14.165955Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T23:49:14.165966Z info JWT policy is third-party-jwt 2026-11-02T23:49:14.165968Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T23:49:14.167274Z info Starting default Istio SDS Server 2026-11-02T23:49:14.167296Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T23:49:14.167315Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T23:49:14.167755Z info Opening status port 15020 2026-11-02T23:49:14.168227Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T23:49:14.168582Z info sds Starting SDS grpc server 2026-11-02T23:49:14.168596Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T23:49:14.170276Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T23:49:14.172282Z info Starting proxy agent 2026-11-02T23:49:14.172312Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T23:49:14.239684Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T23:49:14.248200Z info cache generated new workload certificate resourceName=default latency=79.208676ms ttl=23h59m59.751803074s 2026-11-02T23:49:14.248245Z info cache Root cert has changed, start rotating root cert 2026-11-02T23:49:14.248284Z info cache returned workload trust anchor from cache ttl=23h59m59.751717641s 2026-11-02T23:49:14.465689Z info ads ADS: new connection for node:1 2026-11-02T23:49:14.465767Z info cache returned workload certificate from cache ttl=23h59m59.534236374s 2026-11-02T23:49:14.465831Z info ads ADS: new connection for node:2 2026-11-02T23:49:14.465930Z info cache returned workload trust anchor from cache ttl=23h59m59.534071961s 2026-11-02T23:49:14.565357Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:49:22.893570Z info Readiness succeeded in 8.73182715s 2026-11-02T23:49:22.906047Z info Envoy proxy is ready 2026-11-02T23:53:58.909689Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:54:26.560698Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:54:56.449730Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T23:56:20.199219Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:11:19.428539Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:11:19.428580Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:11:19.428596Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:11:19.428601Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T00:11:19.242Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 30 314 186 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "c478d638-5f52-416a-a959-c3299f986d6b" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.120:52462 10.133.0.120:443 10.132.0.13:34586 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:11:24.759182Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:11:24.759231Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T00:11:24.759255Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T00:11:24.759262Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T00:11:24.650Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 108 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "89cdf4c8-3e19-483c-8ef3-3769a636d8f0" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.120:52462 10.133.0.120:443 10.132.0.13:35260 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:11:27.350923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:11:52.191596Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:12:02.331072Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:12:02.331108Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:12:02.331121Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:12:02.331126Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T00:12:02.207Z] "POST /v1/api-keys HTTP/1.1" 400 - via_upstream - "-" 80 94 123 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "964d3b7a-8857-4e62-812c-c2ad46a96cd4" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.120:44326 10.133.0.120:443 10.132.0.13:35106 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T00:12:30.063121Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:12:33.628Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 5518 5791 85 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "b792584b-512f-4ee0-935b-c75f9487a89d" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.134.0.41:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:38148 10.133.0.120:443 10.132.0.13:49568 - v1-chat-completions-path [2026-11-03T00:12:33.920Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 108118 108394 338 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "6d5455bb-fa40-4b43-babe-1c98269ae46c" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.134.0.41:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:38158 10.133.0.120:443 10.132.0.13:49576 - v1-chat-completions-path [2026-11-03T00:12:34.920Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 140 405 61 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "f347c5c9-e926-4ef3-b16a-bec61d4d43a5" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.134.0.41:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:38148 10.133.0.120:443 10.132.0.13:49584 - v1-chat-completions-path 2026-11-03T00:12:48.435783Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:12:52.825089Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:13:16.612266Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:13:19.732229Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T00:13:19.733671Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 2026-11-03T00:13:20.179754Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-03T00:13:20.186322Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 2026-11-03T00:13:20.583270Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T00:13:19.647Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 85 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "e41c5c49-de57-40cf-ab9a-a3f6f87c17c6" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:56228 10.133.0.120:443 10.132.0.13:49332 - v1-catch-all-path [2026-11-03T00:13:20.099Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 87 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "d00509e8-6c07-43d2-aaee-cb5da79b54c6" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:56236 10.133.0.120:443 10.132.0.13:57482 - v1-catch-all-path [2026-11-03T00:13:20.493Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 89 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "5ed9f2d2-7fff-4a22-8e85-e66e02f79578" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:56228 10.133.0.120:443 10.132.0.13:57484 - v1-catch-all-path 2026-11-03T00:13:34.487637Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:13:39.101436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:13:41.866309Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T00:13:41.794Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 71 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "50c02b44-ba90-4c10-a64d-6f99d2a35458" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:56228 10.133.0.120:443 10.132.0.13:36932 - v1-catch-all-path 2026-11-03T00:13:55.453742Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:14:07.146099Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:14:09.228342Z info ads ADS: new connection for node:3 2026-11-03T00:14:09.228555Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T00:14:09.228589Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T00:14:13.630762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:14:53.156512Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:15:16.942312Z info ads ADS: "" 3 terminated 2026-11-03T00:15:17.272693Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:15:23.450169Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:15:39.294299Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:16:57.752621Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-03T00:18:36.992825Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:19:04.287646Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:19:32.127539Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:19:47.366276Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:19:59.911968Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:20:11.141800Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:20:25.417224Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:20:55.212494Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:21:12.709Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 283 78 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "b3ca11be-58d9-4722-8539-278163070f8c" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:53176 10.133.0.120:443 10.132.0.13:48536 - v1-completions-path 2026-11-03T00:21:43.206698Z info ads ADS: new connection for node:4 2026-11-03T00:21:43.213185Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T00:21:43.213230Z info cache read certificate from file resource=file-root:system 2026-11-03T00:21:47.120309Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:21:47.972916Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:21:52.978173Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"503","StreamId":"4691399125581180864"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-03T00:21:52.906Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 71 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "7dcfc84e-81b7-48d8-8510-ff19d1602b70" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.120:443 10.132.0.13:37046 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T00:23:08.992504Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:23:08.992536Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:23:08.992546Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:23:08.992551Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T00:23:08.992711Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T00:23:08.791Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1716 201 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "7d266ad4-6ad7-42e1-adf7-b6ecd30e42e9" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.120:52298 10.133.0.120:443 10.132.0.13:52240 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T00:23:20.803771Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:23:20.803804Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:23:20.803814Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T00:23:20.803818Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T00:23:20.804005Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T00:23:20.672Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1719 131 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "dbe66252-a072-4d2c-89d9-c5518aa59958" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.120:52298 10.133.0.120:443 10.132.0.13:59592 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T00:23:43.187808Z info ads ADS: "" 4 terminated 2026-11-03T00:23:43.231601Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:23:45.588699Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:23:47.370917Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T00:23:47.370940Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T00:23:47.370943Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-03T00:23:47.370947Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-03T00:23:47.314Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 56 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "41d85c94-066a-4006-9093-d144123cf38a" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.120:443 10.132.0.13:53110 - v1-completions-path 2026-11-03T00:23:55.531465Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:23:58.869Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 80 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "42750a24-e41f-41f3-93a1-d59e86702f91" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:40332 10.133.0.120:443 10.132.0.13:57038 - v1-completions-path 2026-11-03T00:24:13.772890Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:24:19.957511Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:24:23.876Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 68 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "a1b5790f-edc0-4363-83da-310d1e48d051" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:42312 10.133.0.120:443 10.132.0.13:51634 - v1-completions-path 2026-11-03T00:24:40.134820Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:25:09.029172Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:25:17.862Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 61 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "bbfca9e1-4b79-4b38-ad2c-ce3f02ea8b23" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.120:443 10.132.0.13:57498 - v1-completions-path [2026-11-03T00:25:18.648Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "87299517-d57e-4f3c-928f-5b13405bc2c2" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:40332 10.133.0.120:443 10.132.0.13:57512 - v1-completions-path 2026-11-03T00:25:39.548938Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:25:50.418438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:25:56.539147Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:26:10.740Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 64 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "c47a1a69-ca30-4a3c-b25d-7a73bde7d438" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:40332 10.133.0.120:443 10.132.0.13:53564 - v1-completions-path 2026-11-03T00:26:25.301733Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:26:29.806229Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:26:42.662432Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:26:45.537Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 70 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "f4e57a6f-a4b3-42cf-bfe2-1fa5e93dfd48" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:53176 10.133.0.120:443 10.132.0.13:34952 - v1-completions-path 2026-11-03T00:26:51.402965Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:27:11.896112Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:27:37.649233Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:27:50.729322Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:27:56.036Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 66 18 59 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "18103802-8c7e-4763-b588-50f7379b6019" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.120:443 10.132.0.13:40234 - v1-completions-path 2026-11-03T00:27:56.683726Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:28:14.449416Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:28:19.896645Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:28:32.443981Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:00.940192Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:11.740458Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:32.732064Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:29:39.944138Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:32:03.941269Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:33:32.720302Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:33:50.085194Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:33:55.048240Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:08.346103Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:24.118218Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:34:32.171951Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:34:37.841Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 66 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "e0dea22e-a95c-45f9-8b25-484a39532f8d" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:42312 10.133.0.120:443 10.132.0.13:35520 - v1-completions-path 2026-11-03T00:34:54.397074Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:15.060324Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:21.729172Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:50.587319Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:35:56.964065Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:36:03.125855Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:36:17.852Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 75 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "8caa6c87-d5fd-485a-a2ca-d3277b700cab" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:42312 10.133.0.120:443 10.132.0.13:55024 - v1-completions-path [2026-11-03T00:36:20.687Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 68 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "853ad283-c61a-4117-9d4a-0d3a935a6c36" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:40332 10.133.0.120:443 10.132.0.13:47834 - v1-completions-path 2026-11-03T00:36:35.364896Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:36:46.008149Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:37:01.627897Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:37:09.594477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T00:37:12.548Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 67 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "6f08ecc7-3f32-4e25-b054-fa46cade3e49" "maas.apps.ad05c5333c701bf13b6d.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.120:40332 10.133.0.120:443 10.132.0.13:50480 - v1-completions-path 2026-11-03T00:37:16.084896Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:37:39.456133Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:37:50.043911Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:37:55.038459Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:38:12.496724Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-11-03T00:38:13.217495Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:38:19.564894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:38:43.113473Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:39:03.308533Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:39:15.850339Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:39:35.764567Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:39:46.482653Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:39:57.473551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:40:08.182183Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:40:18.851712Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:40:29.737996Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T00:40:40.301396Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8