2026-10-30T05:16:59.674771Z info FLAG: --concurrency="0" 2026-10-30T05:16:59.674788Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T05:16:59.674792Z info FLAG: --help="false" 2026-10-30T05:16:59.674795Z info FLAG: --log_as_json="false" 2026-10-30T05:16:59.674797Z info FLAG: --log_caller="" 2026-10-30T05:16:59.674799Z info FLAG: --log_output_level="default:info" 2026-10-30T05:16:59.674801Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T05:16:59.674805Z info FLAG: --log_target="[stdout]" 2026-10-30T05:16:59.674808Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T05:16:59.674809Z info FLAG: --outlierLogPath="" 2026-10-30T05:16:59.674811Z info FLAG: --profiling="true" 2026-10-30T05:16:59.674813Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T05:16:59.674815Z info FLAG: --proxyLogLevel="warning" 2026-10-30T05:16:59.674817Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T05:16:59.674819Z info FLAG: --stsPort="0" 2026-10-30T05:16:59.674821Z info FLAG: --templateFile="" 2026-10-30T05:16:59.674823Z info FLAG: --tokenManagerPlugin="" 2026-10-30T05:16:59.674825Z info FLAG: --vklog="0" 2026-10-30T05:16:59.674828Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T05:16:59.674963Z info Proxy role ips=[10.133.1.10] type=router id=maas-default-gateway-openshift-default-7f694df8bc-mfjwn.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T05:16:59.675005Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T05:16:59.676264Z info cpu limit detected as 2, setting concurrency 2026-10-30T05:16:59.676505Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T05:16:59.676517Z info JWT policy is third-party-jwt 2026-10-30T05:16:59.676520Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T05:16:59.677809Z info Starting default Istio SDS Server 2026-10-30T05:16:59.677842Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T05:16:59.677861Z info Opening status port 15020 2026-10-30T05:16:59.677870Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T05:16:59.679601Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T05:16:59.681181Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T05:16:59.682005Z info Starting proxy agent 2026-10-30T05:16:59.682040Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T05:16:59.690774Z info sds Starting SDS grpc server 2026-10-30T05:16:59.690794Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T05:16:59.743524Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T05:16:59.756588Z info cache generated new workload certificate resourceName=default latency=76.469213ms ttl=23h59m59.243415718s 2026-10-30T05:16:59.756627Z info cache Root cert has changed, start rotating root cert 2026-10-30T05:16:59.756662Z info cache returned workload trust anchor from cache ttl=23h59m59.243338403s 2026-10-30T05:16:59.855982Z info ads ADS: new connection for node:1 2026-10-30T05:16:59.856375Z info cache returned workload certificate from cache ttl=23h59m59.143627619s 2026-10-30T05:16:59.856670Z info ads ADS: new connection for node:2 2026-10-30T05:16:59.857151Z info cache returned workload trust anchor from cache ttl=23h59m59.142851821s 2026-10-30T05:16:59.943605Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:17:07.793454Z info Readiness succeeded in 8.121275779s 2026-10-30T05:17:07.793771Z info Envoy proxy is ready 2026-10-30T05:17:23.797092Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:17:27.332Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 108118 108394 147 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "87517e49-f54d-4d2a-9ddb-478977cbcfa5" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.30:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:49524 10.133.1.10:443 10.132.0.13:42258 - v1-chat-completions-path 2026-10-30T05:17:39.805436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:17:42.372693Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:17:44.987714Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:17:44.987749Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:17:44.987762Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T05:17:44.987767Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:17:45.478766Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-10-30T05:17:44.892Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 95 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "1a03544c-3537-4e17-81b9-25fc617d8b50" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.10:60262 10.133.1.10:443 10.132.0.13:59412 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T05:17:45.374Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 104 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "1d53ec02-6e43-4196-b50d-05eff88bdc82" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.28:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:35728 10.133.1.10:443 10.132.0.13:34734 - v1-catch-all-path 2026-10-30T05:17:48.717838Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:17:55.986583Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:18:05.775573Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:18:08.546229Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-10-30T05:18:08.546999Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-10-30T05:18:08.486Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 60 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "b7c69a90-263f-4b07-9117-a6a792212111" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.28:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:35728 10.133.1.10:443 10.132.0.13:36178 - v1-catch-all-path 2026-10-30T05:18:20.553038Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:18:31.451335Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:18:33.474951Z info ads ADS: new connection for node:3 2026-10-30T05:18:33.475131Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T05:18:33.475169Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T05:18:35.528068Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:19:15.836280Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:19:34.960297Z info ads ADS: "" 3 terminated 2026-10-30T05:19:35.008358Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:19:37.371Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 283 73 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "2e41a92a-2338-47b4-a68f-b93355c7e204" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:59212 10.133.1.10:443 10.132.0.13:52666 - v1-completions-path 2026-10-30T05:19:38.391099Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:19:46.211388Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:21:20.011693Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:21:37.162870Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:21:57.301764Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:22:11.606090Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:22:16.469964Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:22:20.376092Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:22:30.682874Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:22:35.452116Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:22:50.858933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:23:36.927Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 70 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "525f2a1e-f370-4a3e-8242-4cd6987911d8" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:59212 10.133.1.10:443 10.132.0.13:50496 - v1-completions-path 2026-10-30T05:23:38.386761Z info ads ADS: new connection for node:4 2026-10-30T05:23:38.393750Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T05:23:38.393801Z info cache read certificate from file resource=file-root:system 2026-10-30T05:23:39.486783Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:23:41.517055Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:23:48.644925Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"134","StreamId":"14086785751832270059"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-10-30T05:23:48.582Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 63 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "8b6ef3a9-a8a7-4751-b176-5b0777338402" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.1.10:443 10.132.0.13:53100 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T05:24:12.124746Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:12.124782Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:12.124794Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T05:24:12.124799Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:12.125002Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T05:24:12.017Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1726 107 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "e99492e2-ae68-4c6d-82ec-6e6e31d30115" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.10:53738 10.133.1.10:443 10.132.0.13:41404 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T05:24:40.017850Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:24:40.017884Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:24:40.017895Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:24:40.017899Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T05:24:40.018048Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T05:24:39.916Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1723 101 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "3cc24bc6-2419-4e3d-964b-81775b6d81c0" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.10:52598 10.133.1.10:443 10.132.0.13:43972 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T05:25:01.330106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:25:01.330140Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:25:01.330152Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:25:01.330156Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T05:25:01.330305Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T05:25:01.231Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 98 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "cb877695-9dfc-45cd-a9e1-42c82d0185fb" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.1.10:53738 10.133.1.10:443 10.132.0.13:45150 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T05:25:05.298085Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:25:05.298122Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:25:05.298137Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T05:25:05.298142Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T05:25:05.222Z] "DELETE /v1/api-keys/a41b0746-20b8-47db-ae6a-8f544024a864 HTTP/1.1" 200 - via_upstream - "-" 0 359 75 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "a3f1c9b3-d986-440b-964b-a56129961425" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.10:33062 10.133.1.10:443 10.132.0.13:45152 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:25:16.986210Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:25:17.131227Z info ads ADS: "" 4 terminated with stream closed 2026-10-30T05:25:18.584295Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:25:25.695335Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:25:28.704Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 75 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "95009790-a34d-4a13-914a-c5931d245960" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:38604 10.133.1.10:443 10.132.0.13:37496 - v1-completions-path 2026-10-30T05:25:41.485025Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:25:46.193465Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:25:49.411080Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-10-30T05:25:49.349Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 68 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "aaace23d-338e-4e60-9e75-8e8315b7d2eb" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:59266 10.133.1.10:443 10.132.0.13:37202 - v1-completions-path [2026-10-30T05:25:49.447Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 54 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "f9145cfb-c49f-46f0-8d51-9ae176cfea42" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.10:443 10.132.0.13:37208 - v1-completions-path 2026-10-30T05:26:01.880433Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:26:17.816257Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:26:41.461263Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:26:51.758730Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:26:56.446385Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:27:13.390038Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:27:15.925616Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:27:23.381996Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:27:30.470Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 63 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "d6aed491-35ef-48c2-8516-e95c88655aab" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:50856 10.133.1.10:443 10.132.0.13:60970 - v1-completions-path 2026-10-30T05:27:33.011573Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:27:38.425540Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:27:46.619940Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:27:51.886116Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:28:10.966396Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:28:13.695095Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:28:13.695171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:28:13.695193Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T05:28:13.695199Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:28:13.981159Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:28:13.585Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 324 109 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "c1aafd72-3aaf-494c-b4b2-eeb497c40366" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.10:59850 10.133.1.10:443 10.132.0.13:47970 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T05:28:27.280Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 37 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "863cd09c-7699-4fb6-b277-cd018f4106fe" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.10:443 10.132.0.13:40774 - v1-completions-path 2026-10-30T05:28:29.495270Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:28:30.217184Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:28:30.217223Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:28:30.217238Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T05:28:30.217244Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T05:28:30.141Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 89 332 76 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "588be7e1-aaeb-4599-bb13-d05f4911509d" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.10:41480 10.133.1.10:443 10.132.0.13:40786 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:28:32.370107Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:28:32.316Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 66 18 104 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "986da7fa-3faf-448e-8df1-442b6f3a14c1" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.10:443 10.132.0.13:40804 - v1-completions-path 2026-10-30T05:28:41.526978Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:28:47.233Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 297 63 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "4cb02662-7006-4743-b22a-74e3045dac01" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:59266 10.133.1.10:443 10.132.0.13:37358 - v1-completions-path 2026-10-30T05:28:59.545923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:29:09.904742Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:29:24.479Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 36 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "bc6cb726-1bb7-4b90-b3da-7116368909a3" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.10:443 10.132.0.13:34524 - v1-completions-path 2026-10-30T05:29:26.561675Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:29:30.089014Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:31:01.941433Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:31:05.626103Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:32:31.619938Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:32:46.800295Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:32:49.342019Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:32:49.504Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 106 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "3daa6226-3e43-4662-9d89-9059e7fa7193" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:50856 10.133.1.10:443 10.132.0.13:35170 - v1-completions-path 2026-10-30T05:32:56.750139Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:33:13.665514Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:33:16.725908Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:33:32.296694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:33:43.502295Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:33:48.313368Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:33:52.482Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 56 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "c16b345c-3f5a-46e3-8658-d0d496fee0d2" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:38604 10.133.1.10:443 10.132.0.13:40022 - v1-completions-path [2026-10-30T05:33:52.660Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 56 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "1bb1f69c-be48-4474-b5bc-7b5c08dff815" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:38604 10.133.1.10:443 10.132.0.13:40026 - v1-completions-path 2026-10-30T05:34:05.313211Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:34:10.077048Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:34:13.306177Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:34:20.075Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 59 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "bf3d0eb9-76fb-4086-a8a8-5ed149a697b7" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:38604 10.133.1.10:443 10.132.0.13:52606 - v1-completions-path [2026-10-30T05:34:22.747Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 60 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "c2988150-7c5c-4d6e-8d32-9bc6086311eb" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:38604 10.133.1.10:443 10.132.0.13:52614 - v1-completions-path 2026-10-30T05:34:36.330676Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:34:46.714690Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:34:51.568738Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:34:54.603649Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:34:59.698250Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:35:04.863Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "9f1f6836-8cbe-4618-a8da-e279cb208476" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:38604 10.133.1.10:443 10.132.0.13:43852 - v1-completions-path [2026-10-30T05:35:05.104Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 58 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "d0505cf1-fa8c-4f86-9826-000cf3c89d44" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:38604 10.133.1.10:443 10.132.0.13:43868 - v1-completions-path 2026-10-30T05:35:18.044295Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:35:28.384320Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:35:30.886726Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:35:40.659401Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:35:45.280279Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:36:02.069922Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:36:22.014453Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:36:32.386221Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:36:42.858039Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:36:53.354779Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:37:03.705046Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:37:14.099482Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:37:24.434025Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:37:34.780639Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:37:45.149537Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:37:55.527101Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:38:02.708Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 288 77 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "f10c7d2d-0749-43c3-b84d-a920c830a4c2" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.10:59212 10.133.1.10:443 10.132.0.13:47838 - v1-completions-path