2026-10-30T03:47:09.032819Z info FLAG: --concurrency="0" 2026-10-30T03:47:09.032840Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T03:47:09.032845Z info FLAG: --help="false" 2026-10-30T03:47:09.032848Z info FLAG: --log_as_json="false" 2026-10-30T03:47:09.032852Z info FLAG: --log_caller="" 2026-10-30T03:47:09.032855Z info FLAG: --log_output_level="default:info" 2026-10-30T03:47:09.032857Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T03:47:09.032864Z info FLAG: --log_target="[stdout]" 2026-10-30T03:47:09.032867Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T03:47:09.032871Z info FLAG: --outlierLogPath="" 2026-10-30T03:47:09.032874Z info FLAG: --profiling="true" 2026-10-30T03:47:09.032877Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T03:47:09.032879Z info FLAG: --proxyLogLevel="warning" 2026-10-30T03:47:09.032882Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T03:47:09.032885Z info FLAG: --stsPort="0" 2026-10-30T03:47:09.032888Z info FLAG: --templateFile="" 2026-10-30T03:47:09.032890Z info FLAG: --tokenManagerPlugin="" 2026-10-30T03:47:09.032894Z info FLAG: --vklog="0" 2026-10-30T03:47:09.032897Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T03:47:09.033077Z info Proxy role ips=[10.133.0.244] type=router id=maas-default-gateway-openshift-default-77bfb58f8c-fs5rj.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T03:47:09.033139Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T03:47:09.034208Z info cpu limit detected as 2, setting concurrency 2026-10-30T03:47:09.034456Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T03:47:09.034468Z info JWT policy is third-party-jwt 2026-10-30T03:47:09.034472Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T03:47:09.035963Z info Starting default Istio SDS Server 2026-10-30T03:47:09.035987Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T03:47:09.036006Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T03:47:09.036824Z info Opening status port 15020 2026-10-30T03:47:09.037350Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T03:47:09.039308Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T03:47:09.040350Z info Starting proxy agent 2026-10-30T03:47:09.040389Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T03:47:09.048496Z info sds Starting SDS grpc server 2026-10-30T03:47:09.048515Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T03:47:09.123652Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T03:47:09.308566Z info cache generated new workload certificate resourceName=default latency=271.460753ms ttl=23h59m59.691441514s 2026-10-30T03:47:09.308692Z info cache Root cert has changed, start rotating root cert 2026-10-30T03:47:09.308774Z info cache returned workload trust anchor from cache ttl=23h59m59.691227429s 2026-10-30T03:47:09.378449Z info ads ADS: new connection for node:1 2026-10-30T03:47:09.383579Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T03:47:09.383626Z info cache read certificate from file resource=file-root:system 2026-10-30T03:47:09.471481Z info ads ADS: new connection for node:2 2026-10-30T03:47:09.471561Z info cache returned workload certificate from cache ttl=23h59m59.528441231s 2026-10-30T03:47:09.472919Z info ads ADS: new connection for node:3 2026-10-30T03:47:09.473098Z info cache returned workload trust anchor from cache ttl=23h59m59.52690395s 2026-10-30T03:47:09.582198Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:47:20.268891Z info Readiness succeeded in 11.238528671s 2026-10-30T03:47:20.269336Z info Envoy proxy is ready 2026-10-30T03:47:20.358874Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:47:31.528776Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:47:35.419306Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:47:40.055287Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:47:42.274680Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:47:47.881344Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:47:49.106004Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:48:17.242618Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:49:43.834933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:50:10.312101Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:50:20.148571Z info ads ADS: new connection for node:4 2026-10-30T03:50:20.148765Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T03:50:20.148789Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T03:50:26.452759Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:50:45.820813Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:52:49.201317Z info ads ADS: "" 4 terminated with stream closed 2026-10-30T03:53:15.064541Z info ads ADS: "" 1 terminated with stream closed 2026-10-30T03:57:19.918824Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T03:58:41.950797Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:00:41.707540Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:01:11.203992Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:02:45.447769Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:11:58.910694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:12:20.481033Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:12:30.974662Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:12:30.974698Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:12:30.974710Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T04:12:30.974715Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T04:12:30.885Z] "POST /v1/api-keys HTTP/1.1" 400 - via_upstream - "-" 80 94 89 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "1a0efafd-34fe-49df-881c-64300f6b4531" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.244:36832 10.133.0.244:443 10.133.0.13:38368 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:12:54.924143Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:12:57.942Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 108118 108394 162 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "9d578ad0-7798-93e2-a830-136ada4fb139" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.134.0.31:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.244:48182 10.133.0.244:443 10.133.0.13:56166 - v1-chat-completions-path [2026-10-30T04:12:58.137Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 5518 5791 65 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "99c4278b-f3dd-4474-a53c-a4c531881efb" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.134.0.31:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.244:48182 10.133.0.244:443 10.133.0.13:56178 - v1-chat-completions-path 2026-10-30T04:13:10.664322Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:13:13.215230Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:13:19.737021Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:13:27.253900Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:13:36.496585Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:13:39.126090Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:13:39.126127Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:13:39.126142Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:13:39.126147Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T04:13:39.024Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 101 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "892c2fe3-de8a-4775-aee1-0e447e8bb209" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.244:35858 10.133.0.244:443 10.133.0.13:45562 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:13:51.212754Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:14:02.159560Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:14:04.013619Z info ads ADS: new connection for node:5 2026-10-30T04:14:04.013906Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T04:14:06.086919Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:14:46.833864Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:14:47.714Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 334 2011 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "5c7aa7e6-56ca-4ae8-bde9-564e9af5a11e" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.1.90:8000" outbound|54321||e2e-prio-pool-f2b786ee-inference-pool-ip-19c7a697.llm.svc.cluster.local 10.133.0.244:49364 10.133.0.244:443 10.133.0.13:52908 - v1-model-routing 2026-10-30T04:14:50.489953Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-30T04:14:50.426Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 327 68 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "274d8759-c378-4b25-87e8-3aba466a5081" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.1.90:8000" outbound|54321||e2e-prio-pool-f2b786ee-inference-pool-ip-19c7a697.llm.svc.cluster.local 10.133.0.244:49364 10.133.0.244:443 10.133.0.13:42634 - v1-model-routing [2026-10-30T04:14:53.078Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 336 61 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "4a39a826-1968-4490-b2bf-7abd2fa6122c" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.1.90:8000" outbound|54321||e2e-prio-pool-f2b786ee-inference-pool-ip-19c7a697.llm.svc.cluster.local 10.133.0.244:49364 10.133.0.244:443 10.133.0.13:42650 - v1-model-routing 2026-10-30T04:15:05.698873Z info ads ADS: "" 5 terminated 2026-10-30T04:15:05.745620Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:15:09.323419Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:15:17.434585Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:16:45.020297Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:17:01.344117Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:17:17.494984Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:17:29.349632Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:17:33.370743Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:17:44.401061Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:17:47.965409Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:18:03.012279Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:18:50.131717Z info ads ADS: new connection for node:6 2026-10-30T04:18:50.136995Z info cache read certificate from file resource=file-root:system 2026-10-30T04:18:51.242012Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:18:52.951088Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:18:59.970458Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"521","StreamId":"3525435313701450807"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-10-30T04:18:59.903Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 67 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "6671f754-f12e-44fd-b5b5-897c92c7eaca" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.244:443 10.133.0.13:54596 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T04:19:02.049694Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"525","StreamId":"2322596967934387900"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-10-30T04:19:01.994Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 55 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "eeffd182-ec4e-4a4b-bdb1-aefddb547691" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.244:443 10.133.0.13:37942 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T04:19:11.047337Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T04:20:40.560364Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:20:40.665276Z info ads ADS: "" 6 terminated 2026-10-30T04:20:40.902355Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:20:40.902391Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:20:40.902405Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:20:40.902410Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T04:20:40.820Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 74 317 82 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "65c387f2-a63b-4e94-bd85-04ce3460d3e1" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.244:50736 10.133.0.244:443 10.133.0.13:42952 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:20:42.055930Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:20:48.788563Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:20:51.496072Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-10-30T04:20:51.489Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 73 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "b588ec4a-cf8b-46c0-b5ae-cde4def7faaf" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.244:38492 10.133.0.244:443 10.133.0.13:52996 - v1-completions-path 2026-10-30T04:21:04.521775Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:21:09.321381Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:21:26.193083Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:21:40.465785Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:21:45.617218Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:21:45.617255Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:21:45.617269Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T04:21:45.617274Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T04:21:45.544Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 77 320 73 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "ef83e75f-0aec-455b-a9ad-decf17f6447b" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.244:50540 10.133.0.244:443 10.133.0.13:40466 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T04:21:47.232Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "af12ce72-11d6-416a-ae71-6db00eec04ca" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.244:42100 10.133.0.244:443 10.133.0.13:40482 - v1-completions-path 2026-10-30T04:22:03.909987Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:22:14.136052Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:22:18.927734Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:22:22.663046Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:22:22.663086Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:22:22.663103Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:22:22.663108Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T04:22:22.574Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 317 88 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "4f20980b-7556-4b62-94c6-2a69ea17f18e" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.244:40926 10.133.0.244:443 10.133.0.13:43988 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T04:22:22.772Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "14f97002-1974-4abb-843e-1e5b9e6eb6f5" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.244:42100 10.133.0.244:443 10.133.0.13:44004 - v1-completions-path [2026-10-30T04:22:23.230Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 61 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "84cf3e55-141a-4fbb-82c5-f550c913ef11" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.244:38492 10.133.0.244:443 10.133.0.13:44006 - v1-completions-path [2026-10-30T04:22:23.659Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 62 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "56295513-9b5a-4733-ba60-bfd5e04b2c7d" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.244:38492 10.133.0.244:443 10.133.0.13:44022 - v1-completions-path 2026-10-30T04:22:36.022078Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:22:38.434527Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:22:45.763332Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:22:57.555890Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:23:02.891492Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:23:10.994237Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:23:16.388019Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:23:34.668681Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:23:37.759636Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:23:53.274372Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:23:56.190331Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:23:57.207Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 75 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "6f0a8305-271c-4f4e-ae5b-b81ecb3bdedc" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.0.53:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.244:46084 10.133.0.244:443 10.133.0.13:56700 - v1-completions-path 2026-10-30T04:24:04.252117Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:24:22.220874Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:24:32.589918Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:24:49.975389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:24:53.487036Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:26:11.662153Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:26:11.662189Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:26:11.662206Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:26:11.662211Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T04:26:11.580Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 80 323 82 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "8c793197-6962-4f0d-a63c-bb73dff92a09" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.244:56840 10.133.0.244:443 10.133.0.13:56666 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:26:24.635657Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:26:28.002499Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:27:53.188477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:28:08.546830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:28:10.929711Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:28:18.098551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:28:38.350156Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:28:41.435913Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:28:57.005383Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:29:07.340750Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:29:12.479881Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:29:31.206070Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:29:35.860012Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:29:38.740793Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:29:46.189Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 65 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "36562c8a-4907-4ae6-8be8-2e36c4cb883f" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.244:42100 10.133.0.244:443 10.133.0.13:55972 - v1-completions-path [2026-10-30T04:29:47.291Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 57 - "34.228.250.194,10.133.0.13" "python-requests/2.34.2" "a19e7900-72bf-41f7-8784-e69d3104620e" "maas.apps.1ef8b236caec72cd892e.prod.konflux-ocp-ci.dev" "10.133.0.52:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.244:42100 10.133.0.244:443 10.133.0.13:55988 - v1-completions-path 2026-10-30T04:30:01.852500Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:30:12.222155Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:30:14.717538Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:30:17.642774Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:30:22.841946Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:30:41.133489Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:30:51.476468Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:30:53.930578Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:31:03.662536Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:31:08.561681Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:31:25.369921Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:31:45.228156Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:31:55.549468Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:32:05.903519Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:32:16.333296Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:32:26.587696Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:32:36.937364Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:32:47.296715Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:32:57.704182Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:33:07.979993Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:33:18.249630Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8