2026-10-30T07:34:48.002763Z info FLAG: --concurrency="0" 2026-10-30T07:34:48.002781Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T07:34:48.002785Z info FLAG: --help="false" 2026-10-30T07:34:48.002787Z info FLAG: --log_as_json="false" 2026-10-30T07:34:48.002790Z info FLAG: --log_caller="" 2026-10-30T07:34:48.002792Z info FLAG: --log_output_level="default:info" 2026-10-30T07:34:48.002794Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T07:34:48.002798Z info FLAG: --log_target="[stdout]" 2026-10-30T07:34:48.002800Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T07:34:48.002802Z info FLAG: --outlierLogPath="" 2026-10-30T07:34:48.002804Z info FLAG: --profiling="true" 2026-10-30T07:34:48.002806Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T07:34:48.002808Z info FLAG: --proxyLogLevel="warning" 2026-10-30T07:34:48.002810Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T07:34:48.002812Z info FLAG: --stsPort="0" 2026-10-30T07:34:48.002814Z info FLAG: --templateFile="" 2026-10-30T07:34:48.002816Z info FLAG: --tokenManagerPlugin="" 2026-10-30T07:34:48.002819Z info FLAG: --vklog="0" 2026-10-30T07:34:48.002821Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T07:34:48.002991Z info Proxy role ips=[10.133.1.16] type=router id=maas-default-gateway-openshift-default-85544d4cd7-cnmjg.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T07:34:48.003040Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T07:34:48.004032Z info cpu limit detected as 2, setting concurrency 2026-10-30T07:34:48.004576Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T07:34:48.004587Z info JWT policy is third-party-jwt 2026-10-30T07:34:48.004590Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T07:34:48.006246Z info Starting default Istio SDS Server 2026-10-30T07:34:48.006268Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T07:34:48.006285Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T07:34:48.007030Z info Opening status port 15020 2026-10-30T07:34:48.007561Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T07:34:48.008897Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T07:34:48.009720Z info Starting proxy agent 2026-10-30T07:34:48.009750Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T07:34:48.019279Z info sds Starting SDS grpc server 2026-10-30T07:34:48.019302Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T07:34:48.073569Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T07:34:48.080764Z info cache generated new workload certificate resourceName=default latency=73.432527ms ttl=23h59m59.919239519s 2026-10-30T07:34:48.080794Z info cache Root cert has changed, start rotating root cert 2026-10-30T07:34:48.080829Z info cache returned workload trust anchor from cache ttl=23h59m59.919172002s 2026-10-30T07:34:48.189505Z info ads ADS: new connection for node:1 2026-10-30T07:34:48.189566Z info cache returned workload certificate from cache ttl=23h59m59.810435925s 2026-10-30T07:34:48.190142Z info ads ADS: new connection for node:2 2026-10-30T07:34:48.190230Z info cache returned workload trust anchor from cache ttl=23h59m59.809771994s 2026-10-30T07:34:48.269865Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:34:56.093570Z info Readiness succeeded in 8.093310427s 2026-10-30T07:34:56.093922Z info Envoy proxy is ready 2026-10-30T07:35:08.364097Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:35:26.477923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:35:28.753902Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:35:37.161771Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-10-30T07:35:37.078Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 83 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "cc0597e3-9eee-436d-a819-240d1e29424a" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.16:50404 10.133.1.16:443 10.132.0.21:42840 - v1-catch-all-path 2026-10-30T07:35:39.651102Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:35:57.701221Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:36:00.015336Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:36:14.818803Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:36:25.830283Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:36:27.871401Z info ads ADS: new connection for node:3 2026-10-30T07:36:27.871751Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T07:36:27.871835Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T07:36:29.878264Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:37:10.381444Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:37:15.703Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 347 94 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "9c842149-3baa-4ba2-8225-36a05895d235" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.133.1.21:8000" outbound|54321||e2e-prio-pool-38cb65a5-inference-pool-ip-555586cf.llm.svc.cluster.local 10.133.1.16:49856 10.133.1.16:443 10.132.0.21:37874 - v1-model-routing [2026-10-30T07:37:28.032Z] "POST /v1/chat/completions HTTP/1.1" 403 - - "-" 131 0 36 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "32762f75-1233-4828-a21c-ab520a6f4b9a" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" outbound|54321||e2e-prio-pool-38cb65a5-inference-pool-ip-555586cf.llm.svc.cluster.local - 10.133.1.16:443 10.132.0.21:33774 - v1-model-routing 2026-10-30T07:37:29.385014Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:37:29.470302Z info ads ADS: "" 3 terminated with stream closed 2026-10-30T07:37:32.993748Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:37:40.626535Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:39:16.033355Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:39:32.443660Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:39:47.834728Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:39:47.834765Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:39:47.834781Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:39:47.834786Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T07:39:47.836862Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:39:47.743Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 628 91 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "d0f06480-97ee-43a2-8d79-723ff47bff6e" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.16:58244 10.133.1.16:443 10.132.0.21:44886 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T07:40:01.886049Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:40:06.788553Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:40:19.299950Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:40:22.908847Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:40:43.057808Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:41:28.913Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 295 80 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "7db9951d-d54b-404c-9c8e-4242feb08398" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.16:48594 10.133.1.16:443 10.132.0.21:40776 - v1-completions-path 2026-10-30T07:41:30.274474Z info ads ADS: new connection for node:4 2026-10-30T07:41:30.283805Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T07:41:30.283863Z info cache read certificate from file resource=file-root:system 2026-10-30T07:41:31.382797Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:41:32.962714Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:41:43.862732Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"139","StreamId":"12989818698307610333"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-10-30T07:41:43.800Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 62 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "2a19cef2-4fe7-4594-b8fe-3c55a4516240" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.1.16:443 10.132.0.21:38822 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T07:42:03.232403Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:42:03.232445Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:42:03.232458Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:42:03.232465Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T07:42:03.232676Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T07:42:03.122Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1720 110 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "1fd6900c-7603-439f-987c-c9152bfdf878" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.16:43826 10.133.1.16:443 10.132.0.21:47116 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:42:16.024261Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:42:16.024290Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:42:16.024300Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:42:16.024305Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T07:42:16.024467Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T07:42:15.922Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1720 102 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "c4fe0b29-5bb9-4271-b0c7-49946f57f34b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.16:43826 10.133.1.16:443 10.132.0.21:56740 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:42:35.268800Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-10-30T07:42:35.316222Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:42:35.316255Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:42:35.316268Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:42:35.316273Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T07:42:35.316467Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T07:42:35.212Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1725 104 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "895024f0-699a-4983-99b5-2b19677af0e4" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.16:49174 10.133.1.16:443 10.132.0.21:38058 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:42:59.295931Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:42:59.451508Z info ads ADS: "" 4 terminated with stream closed 2026-10-30T07:43:00.876026Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:02.818263Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T07:43:02.818284Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T07:43:02.818289Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T07:43:02.818294Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-30T07:43:02.770Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 47 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "2b676d6a-c0ec-4817-ac85-c7527740ad64" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.16:443 10.132.0.21:57990 - v1-completions-path 2026-10-30T07:43:07.766619Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:24.947078Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:29.471997Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:45.203695Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:01.043646Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:23.611499Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:33.859169Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:38.710429Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:45.429411Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:44:45.429446Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:44:45.429462Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:44:45.429467Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T07:44:45.357Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 318 72 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "c8808035-9d00-4188-85ca-692a7500fdb5" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.16:33120 10.133.1.16:443 10.132.0.21:48008 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T07:44:58.740989Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:01.016669Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:20.545798Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:22.883013Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:41.567464Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:43.857101Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:46:01.720674Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:46:12.380161Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:46:15.299955Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:46:34.592664Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:46:37.405077Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:46:49.989683Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:07.976584Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:18.353694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:38.405041Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:41.778742Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:49:24.885155Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:49:28.379175Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:01.741457Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:51:04.426Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 73 286 80 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "8748b888-a623-4bdf-a824-48629c9f83eb" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.33:8000" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.16:45004 10.133.1.16:443 10.132.0.21:43532 - v1-completions-path 2026-10-30T07:51:17.293693Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:19.779726Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:38.611509Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:50.589765Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:53.552169Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:57.076278Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-10-30T07:51:56.650Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 72 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "ca5edfce-f26e-4502-b484-491a6b271a72" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.16:52548 10.133.1.16:443 10.132.0.21:53760 - v1-completions-path [2026-10-30T07:51:57.022Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 55 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "b3335d9e-c1da-4c6c-99e5-1fa28166db42" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.16:443 10.132.0.21:53770 - v1-completions-path 2026-10-30T07:52:09.453550Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:19.987164Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:24.645804Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:28.484120Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:52:28.484157Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:52:28.484172Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:52:28.484178Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T07:52:28.410Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 309 73 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "907a330e-0f63-4084-be7e-18fb72992cb2" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.133.1.61:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.16:37132 10.133.1.16:443 10.132.0.21:36892 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T07:52:41.866272Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:46.495570Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:49.521374Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:52:57.549Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 62 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "8b43f0f8-f280-4ece-a871-ac2fef9cac88" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.16:52548 10.133.1.16:443 10.132.0.21:44740 - v1-completions-path [2026-10-30T07:52:57.733Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 58 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "15be18fb-a2ff-456e-a370-27fcdef533c5" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.16:52548 10.133.1.16:443 10.132.0.21:44742 - v1-completions-path 2026-10-30T07:53:12.540862Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:22.901376Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:27.464605Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:31.127719Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:36.184756Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:53:40.588Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 75 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "8e288456-589a-400c-a9c8-cc3468b4f1b3" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.16:35858 10.133.1.16:443 10.132.0.21:43620 - v1-completions-path 2026-10-30T07:53:54.315911Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:04.702404Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:07.096302Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:18.455808Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:22.088064Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:39.755899Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:42.147745Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:00.470224Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:10.936099Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:21.066762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:31.442277Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:41.838339Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:52.329940Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:56:02.468574Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:56:12.954141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:56:23.147964Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:56:33.489770Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8