2026-10-30T18:42:21.987117Z info FLAG: --concurrency="0" 2026-10-30T18:42:21.987174Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T18:42:21.987179Z info FLAG: --help="false" 2026-10-30T18:42:21.987182Z info FLAG: --log_as_json="false" 2026-10-30T18:42:21.987185Z info FLAG: --log_caller="" 2026-10-30T18:42:21.987186Z info FLAG: --log_output_level="default:info" 2026-10-30T18:42:21.987189Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T18:42:21.987194Z info FLAG: --log_target="[stdout]" 2026-10-30T18:42:21.987197Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T18:42:21.987199Z info FLAG: --outlierLogPath="" 2026-10-30T18:42:21.987201Z info FLAG: --profiling="true" 2026-10-30T18:42:21.987203Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T18:42:21.987206Z info FLAG: --proxyLogLevel="warning" 2026-10-30T18:42:21.987208Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T18:42:21.987211Z info FLAG: --stsPort="0" 2026-10-30T18:42:21.987213Z info FLAG: --templateFile="" 2026-10-30T18:42:21.987215Z info FLAG: --tokenManagerPlugin="" 2026-10-30T18:42:21.987217Z info FLAG: --vklog="0" 2026-10-30T18:42:21.987220Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T18:42:21.987439Z info Proxy role ips=[10.133.0.37] type=router id=maas-default-gateway-openshift-default-855f9574cf-qstgx.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T18:42:21.987495Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T18:42:21.989514Z info cpu limit detected as 2, setting concurrency 2026-10-30T18:42:21.989813Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T18:42:21.989831Z info JWT policy is third-party-jwt 2026-10-30T18:42:21.989836Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T18:42:21.994544Z info Starting default Istio SDS Server 2026-10-30T18:42:21.994594Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T18:42:21.994630Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T18:42:21.994931Z info Opening status port 15020 2026-10-30T18:42:21.995915Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T18:42:21.997665Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T18:42:21.999068Z info Starting proxy agent 2026-10-30T18:42:21.999139Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T18:42:22.001084Z info sds Starting SDS grpc server 2026-10-30T18:42:22.001196Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T18:42:22.094402Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T18:42:22.248421Z info ads ADS: new connection for node:1 2026-10-30T18:42:22.250194Z info ads ADS: new connection for node:2 2026-10-30T18:42:22.334606Z info cache generated new workload certificate resourceName=default latency=338.175456ms ttl=23h59m59.665400016s 2026-10-30T18:42:22.334660Z info cache Root cert has changed, start rotating root cert 2026-10-30T18:42:22.334724Z info cache returned workload trust anchor from cache ttl=23h59m59.665277083s 2026-10-30T18:42:22.334750Z info cache returned workload trust anchor from cache ttl=23h59m59.665250162s 2026-10-30T18:42:22.334769Z info cache returned workload certificate from cache ttl=23h59m59.665235033s 2026-10-30T18:42:22.335281Z info cache returned workload trust anchor from cache ttl=23h59m59.664720902s 2026-10-30T18:42:22.444236Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T18:42:31.120624Z info Readiness succeeded in 9.137720914s 2026-10-30T18:42:31.121001Z info Envoy proxy is ready 2026-10-30T18:45:27.649663Z info ads ADS: new connection for node:3 2026-10-30T18:45:27.656371Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T18:45:27.656480Z info cache read certificate from file resource=file-root:system 2026-10-30T18:45:28.203621Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T18:45:35.399091Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T18:45:44.845656Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 [2026-10-30T18:45:47.908Z] "POST /llm/e2e-rt-sw-ddb042/anything/e2e-rt-sw-ddb042-client HTTP/1.1" 429 - - "-" 124 18 136 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "96d9f85d-f49a-4815-b527-cde3f804a432" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-ddb042.llm.svc.cluster.local - 10.133.0.37:443 10.132.0.19:42664 - llm.external-model-e2e-rt-sw-ddb042-9dfbb4fd00f7.4 2026-10-30T18:46:02.995940Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T18:46:07.481038Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: gRPC status code is not OK thread=24 [2026-10-30T18:46:04.618Z] "POST /maas-api/v1/api-keys HTTP/1.1" 500 - - "-" 74 23 2864 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "91c911c9-7b04-478b-97d3-ee34d3ec0f9b" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.37:443 10.132.0.19:41118 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T18:46:28.804798Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:28.804847Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:28.804863Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:28.804872Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T18:46:28.805118Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T18:46:28.622Z] "POST /anything/e2e-rt-eq-1e8613-client HTTP/1.1" 200 - via_upstream - "-" 124 1772 182 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "1fa8786e-f20f-487d-9891-49dadc932c20" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:39972 10.133.0.37:443 10.132.0.19:52952 - llm.external-model-e2e-rt-eq-1e8613-4d38146b8aab.0 2026-10-30T18:46:29.870503Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:46:29.870554Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:46:29.870571Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:46:29.870578Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T18:46:29.870887Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T18:46:29.648Z] "POST /anything/e2e-rt-eq-1e8613-client HTTP/1.1" 200 - via_upstream - "-" 124 1776 222 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "8d739d89-7d76-4369-997f-188dd87ecd9a" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:39980 10.133.0.37:443 10.132.0.19:52966 - llm.external-model-e2e-rt-eq-1e8613-4d38146b8aab.0 2026-10-30T18:46:30.414071Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:30.414301Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:30.414349Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:30.414376Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T18:46:30.414847Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T18:46:30.217Z] "POST /anything/e2e-rt-eq-1e8613-client HTTP/1.1" 200 - via_upstream - "-" 124 1766 197 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "066da2ee-c54a-446e-95d8-543e0c404d17" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:39988 10.133.0.37:443 10.132.0.19:52982 - llm.external-model-e2e-rt-eq-1e8613-4d38146b8aab.0 2026-10-30T18:46:33.136967Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:46:33.137003Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:46:33.137013Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:46:33.137018Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T18:46:33.137250Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T18:46:32.996Z] "POST /anything/e2e-rt-eq-1e8613-client HTTP/1.1" 200 - via_upstream - "-" 124 1773 140 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "63ffcdbb-bfcf-4805-8b0f-c8025c95b946" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.37:55810 10.133.0.37:443 10.132.0.19:52998 - llm.external-model-e2e-rt-eq-1e8613-4d38146b8aab.0 2026-10-30T18:46:57.381834Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T18:47:01.481836Z info ads ADS: "" 3 terminated with stream closed 2026-10-30T18:47:01.550117Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T18:47:14.271036Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T18:50:37.116145Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:50:37.116210Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:50:37.116234Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T18:50:37.116242Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T18:50:36.901Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 84 327 214 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "d9a33e44-a126-41db-8b0b-d2f3218de0aa" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:40700 10.133.0.37:443 10.132.0.19:47208 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T18:50:56.173Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 48 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "cbf9f81c-3636-4817-9cf3-1a60d04082d5" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.132.0.19:54946 - v1-completions-path 2026-10-30T18:50:56.962693Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:50:56.962818Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:50:56.962873Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:50:56.962886Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T18:50:56.871Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 91 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "26b33614-60ad-47ba-a383-02562d6ac3c9" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:40700 10.133.0.37:443 10.132.0.19:54952 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T18:50:58.405Z] "POST /maas-api/internal/v1/api-keys/validate HTTP/1.1" 404 NR route_not_found - "-" 31 0 15 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "7c0cdc7e-4fb4-4eec-914c-f1bee65d60ca" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "-" - - 10.133.0.37:443 10.132.0.19:54966 - - 2026-10-30T18:52:59.800034Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T18:55:02.551354Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T19:05:19.641934Z info ads ADS: new connection for node:4 2026-10-30T19:05:19.660027Z info cache read certificate from file resource=file-root:system 2026-10-30T19:13:31.435826Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T19:13:48.200853Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T19:14:26.116868Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T19:14:40.226093Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T19:18:59.308474Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T19:19:01.345119Z info ads ADS: new connection for node:5 2026-10-30T19:19:01.345454Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T19:19:01.345551Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T19:19:04.573674Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T19:19:07.752623Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T19:19:09.431482Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T19:19:10.913542Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T19:19:34.323098Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=10 2026-10-30T19:21:39.819391Z info ads ADS: "" 4 terminated 2026-10-30T19:22:29.244082Z info ads ADS: "" 5 terminated with stream closed