2026-11-02T21:22:11.067307Z info FLAG: --concurrency="0" 2026-11-02T21:22:11.067326Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T21:22:11.067330Z info FLAG: --help="false" 2026-11-02T21:22:11.067332Z info FLAG: --log_as_json="false" 2026-11-02T21:22:11.067335Z info FLAG: --log_caller="" 2026-11-02T21:22:11.067337Z info FLAG: --log_output_level="default:info" 2026-11-02T21:22:11.067339Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T21:22:11.067344Z info FLAG: --log_target="[stdout]" 2026-11-02T21:22:11.067346Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T21:22:11.067348Z info FLAG: --outlierLogPath="" 2026-11-02T21:22:11.067350Z info FLAG: --profiling="true" 2026-11-02T21:22:11.067352Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T21:22:11.067354Z info FLAG: --proxyLogLevel="warning" 2026-11-02T21:22:11.067356Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T21:22:11.067358Z info FLAG: --stsPort="0" 2026-11-02T21:22:11.067360Z info FLAG: --templateFile="" 2026-11-02T21:22:11.067362Z info FLAG: --tokenManagerPlugin="" 2026-11-02T21:22:11.067365Z info FLAG: --vklog="0" 2026-11-02T21:22:11.067367Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T21:22:11.067506Z info Proxy role ips=[10.133.0.210] type=router id=maas-default-gateway-openshift-default-5db77fbd9f-z44pn.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T21:22:11.067544Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T21:22:11.068561Z info cpu limit detected as 2, setting concurrency 2026-11-02T21:22:11.068794Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T21:22:11.068806Z info JWT policy is third-party-jwt 2026-11-02T21:22:11.068809Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T21:22:11.071553Z info Starting default Istio SDS Server 2026-11-02T21:22:11.071580Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T21:22:11.071600Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T21:22:11.071725Z info Opening status port 15020 2026-11-02T21:22:11.072389Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T21:22:11.073896Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T21:22:11.074643Z info Starting proxy agent 2026-11-02T21:22:11.074671Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T21:22:11.084871Z info sds Starting SDS grpc server 2026-11-02T21:22:11.084902Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T21:22:11.144605Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T21:22:11.200705Z info cache generated new workload certificate resourceName=default latency=128.014073ms ttl=23h59m59.799298574s 2026-11-02T21:22:11.200754Z info cache Root cert has changed, start rotating root cert 2026-11-02T21:22:11.200788Z info cache returned workload trust anchor from cache ttl=23h59m59.799212043s 2026-11-02T21:22:11.308095Z info ads ADS: new connection for node:1 2026-11-02T21:22:11.308157Z info cache returned workload certificate from cache ttl=23h59m59.691845083s 2026-11-02T21:22:11.308318Z info ads ADS: new connection for node:2 2026-11-02T21:22:11.308402Z info cache returned workload trust anchor from cache ttl=23h59m59.691599798s 2026-11-02T21:22:11.389818Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:22:19.710243Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:22:20.656839Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-02T21:22:21.530908Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-02T21:22:22.168473Z info Readiness succeeded in 11.103782907s 2026-11-02T21:22:22.168956Z info Envoy proxy is ready 2026-11-02T21:22:22.223414Z info ads ADS: new connection for node:3 2026-11-02T21:22:22.223619Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T21:22:22.223650Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T21:22:22.272084Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:23:05.626588Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:23:29.246790Z info ads ADS: "" 3 terminated with stream closed 2026-11-02T21:23:29.541900Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:23:34.166851Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:23:49.988587Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:26:23.298407Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:26:50.489348Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:27:21.492113Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:27:36.675716Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:27:42.874289Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:27:54.361264Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:27:57.329555Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:28:15.318180Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T21:28:18.063Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 88 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "dfce919e-c2c9-450b-8e66-761c5e5c63c5" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.51:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:58612 10.133.0.210:443 10.132.0.22:44376 - v1-completions-path 2026-11-02T21:29:03.086116Z info ads ADS: new connection for node:4 2026-11-02T21:29:03.089879Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T21:29:03.089924Z info cache read certificate from file resource=file-root:system 2026-11-02T21:29:06.675421Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:29:08.002739Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:29:10.962722Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"131","StreamId":"3410095083297439680"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-02T21:29:10.896Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 66 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "9a449f5d-26ba-4921-863c-1c00924a8e33" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.210:443 10.132.0.22:55184 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-02T21:29:22.551171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:29:22.551208Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:29:22.551218Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T21:29:22.551223Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:29:22.551408Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T21:29:22.354Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1718 196 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "5a61a8c5-92ee-4a63-9a45-26235c959b26" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.210:41100 10.133.0.210:443 10.132.0.22:46606 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T21:29:24.950588Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:29:24.950622Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:29:24.950631Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T21:29:24.950636Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:29:24.950802Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T21:29:24.764Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1726 186 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "3925ab6b-798c-9c0f-b76a-ba9717a24524" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.210:53262 10.133.0.210:443 10.132.0.22:46620 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T21:29:43.896075Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:29:43.896106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:29:43.896123Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:29:43.896129Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T21:29:43.896135Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-02T21:29:43.896281Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T21:29:43.774Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 122 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "b7e64bc1-9a19-4ae7-bdbb-3208238d3860" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.210:53262 10.133.0.210:443 10.132.0.22:41834 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T21:30:12.298271Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:30:12.298307Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:30:12.298319Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T21:30:12.298325Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:30:12.298506Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T21:30:12.172Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 126 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "6a824fa3-0f9a-47a8-a92e-c7ffb117d51f" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.210:41100 10.133.0.210:443 10.132.0.22:49750 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T21:30:19.289435Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:30:19.289472Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:30:19.289485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:30:19.289491Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T21:30:19.289676Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T21:30:19.167Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 122 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "8561db61-5a76-4c33-a17d-8645b7a8bf76" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.210:53262 10.133.0.210:443 10.132.0.22:60250 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T21:30:36.663755Z info ads ADS: "" 4 terminated 2026-11-02T21:30:36.706088Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:30:39.261671Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:30:49.474871Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:31:07.723175Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:31:12.182838Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:31:14.741100Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-02T21:31:14.769349Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:31:14.769399Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:31:14.769425Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:31:14.769433Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T21:31:14.597Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 93 336 172 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "ebb70232-e31c-4f2c-b9f1-f6972e4ca0ea" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.133.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.210:49458 10.133.0.210:443 10.132.0.22:57446 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T21:31:15.773Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 288 74 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "ef53598a-e9ff-4b22-ae15-9b4758c98c08" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36482 10.133.0.210:443 10.132.0.22:57454 - v1-completions-path 2026-11-02T21:31:32.223629Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:31:50.683563Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:31:55.348192Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:31:55.348228Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:31:55.348244Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T21:31:55.348249Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T21:31:55.224Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 123 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "c812bca0-3862-4b13-ae47-314c586fd1d9" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.133.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.210:43686 10.133.0.210:443 10.132.0.22:36342 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T21:32:01.254Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 60 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "9f5b1e75-3c6f-4ca3-ba1a-cd37dbd440fe" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36482 10.133.0.210:443 10.132.0.22:57210 - v1-completions-path 2026-11-02T21:32:19.209676Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:32:29.794824Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:32:36.000694Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T21:32:47.747Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 62 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "83265d1a-657a-4fba-b3e7-248e2a704349" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36194 10.133.0.210:443 10.132.0.22:51020 - v1-completions-path [2026-11-02T21:32:48.013Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 57 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "e4c28da4-d317-4ce5-96f4-0f5901a7ddd1" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36194 10.133.0.210:443 10.132.0.22:51034 - v1-completions-path 2026-11-02T21:33:04.418201Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:33:08.693246Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:33:21.508324Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:33:30.266747Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:33:39.399082Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:33:45.432806Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:33:59.292198Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:34:14.339308Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T21:34:17.252Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 66 18 62 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "1d283e3d-26dc-4b3c-8093-90697367fc5d" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.210:443 10.132.0.22:37662 - v1-completions-path 2026-11-02T21:34:20.012402Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:34:38.004062Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:34:43.231813Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:34:54.072866Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:35:18.024445Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:35:28.766570Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:35:49.092883Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:35:56.509506Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:38:22.159883Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:39:51.803266Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:39:55.498402Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:39:55.498437Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:39:55.498450Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:39:55.498455Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T21:39:55.382Z] "DELETE /v1/api-keys/e2e-trlp-test-key HTTP/1.1" 404 - via_upstream - "-" 0 29 116 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "7200f33f-b2db-4433-8b7e-f060e29bc5bb" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.133.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.210:57230 10.133.0.210:443 10.132.0.22:53784 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T21:40:08.988064Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:40:13.756032Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:40:26.440922Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:40:42.633249Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:40:48.986602Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T21:40:54.402Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 68 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "86e72151-79ac-4b5c-8d4a-bae3eb542719" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36194 10.133.0.210:443 10.132.0.22:46288 - v1-completions-path 2026-11-02T21:41:08.575441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:41:19.271880Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:41:25.401884Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:41:52.667659Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:41:58.827907Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:42:04.553011Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:42:11.593006Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:42:11.593047Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:42:11.593070Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T21:42:11.593077Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T21:42:11.478Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 68 310 114 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "5a1d20c6-fd8c-47f7-9108-005b29eb5c64" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.133.0.227:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.210:47628 10.133.0.210:443 10.132.0.22:42178 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T21:42:13.031Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 60 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "34e4c0ed-a267-410a-9484-69eb826852f5" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36194 10.133.0.210:443 10.132.0.22:42188 - v1-completions-path [2026-11-02T21:42:14.018Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 62 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "df4582f4-c15e-4c7f-94bc-e21fdf15320a" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36194 10.133.0.210:443 10.132.0.22:42206 - v1-completions-path [2026-11-02T21:42:15.079Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 59 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "ef5d2474-957e-46c9-a8cb-d8a56e4a4be4" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36482 10.133.0.210:443 10.132.0.22:42220 - v1-completions-path [2026-11-02T21:42:17.619Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 66 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "f5e1d454-802f-4a4b-9340-ed3f41fd7a61" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36194 10.133.0.210:443 10.132.0.22:50550 - v1-completions-path [2026-11-02T21:42:20.175Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 62 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "a66f605d-7909-4440-9d4c-0c099724d0c2" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36482 10.133.0.210:443 10.132.0.22:50566 - v1-completions-path 2026-11-02T21:42:35.073141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:42:45.873487Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:42:52.040033Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:42:57.916658Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:43:07.140675Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T21:43:13.919Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 61 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "58368499-bf06-47f0-9e72-cbf1f8f9e6c5" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.50:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:36194 10.133.0.210:443 10.132.0.22:42094 - v1-completions-path 2026-11-02T21:43:16.865236Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:43:16.865287Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:43:16.865307Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T21:43:16.865313Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T21:43:16.752Z] "DELETE /v1/api-keys/82381ab9-6a7a-4c2d-ac4a-6a8d65bdbe8b HTTP/1.1" 200 - via_upstream - "-" 0 367 112 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "4152954c-519f-4ce0-95e3-838f473594ac" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.134.0.198:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.210:59010 10.133.0.210:443 10.132.0.22:35784 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T21:43:30.333436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:43:41.083364Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:43:45.669631Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:44:03.882528Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:44:10.787909Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:44:20.091426Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:44:41.638401Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:45:01.702814Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:45:14.299874Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:45:33.875858Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:45:44.676806Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:45:55.361617Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:46:06.073781Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:46:16.859125Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T21:46:27.476158Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T21:46:29.351Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 76 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "a2baf901-6813-4bbd-b3be-165465fbad95" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.51:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:58612 10.133.0.210:443 10.132.0.22:54062 - v1-completions-path [2026-11-02T21:46:36.283Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 72 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "d45028f1-079a-4476-b68c-cd960805e0ed" "maas.apps.6fa4745d83b81ed2e5ae.prod.konflux-ocp-ci.dev" "10.132.0.51:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.210:58612 10.133.0.210:443 10.132.0.22:54092 - v1-completions-path