2026-11-04T05:57:15.528906Z info FLAG: --concurrency="0" 2026-11-04T05:57:15.528923Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-04T05:57:15.528927Z info FLAG: --help="false" 2026-11-04T05:57:15.528929Z info FLAG: --log_as_json="false" 2026-11-04T05:57:15.528931Z info FLAG: --log_caller="" 2026-11-04T05:57:15.528934Z info FLAG: --log_output_level="default:info" 2026-11-04T05:57:15.528936Z info FLAG: --log_stacktrace_level="default:none" 2026-11-04T05:57:15.528940Z info FLAG: --log_target="[stdout]" 2026-11-04T05:57:15.528943Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-04T05:57:15.528945Z info FLAG: --outlierLogPath="" 2026-11-04T05:57:15.528947Z info FLAG: --profiling="true" 2026-11-04T05:57:15.528949Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-04T05:57:15.528951Z info FLAG: --proxyLogLevel="warning" 2026-11-04T05:57:15.528953Z info FLAG: --serviceCluster="istio-proxy" 2026-11-04T05:57:15.528956Z info FLAG: --stsPort="0" 2026-11-04T05:57:15.528958Z info FLAG: --templateFile="" 2026-11-04T05:57:15.528960Z info FLAG: --tokenManagerPlugin="" 2026-11-04T05:57:15.528962Z info FLAG: --vklog="0" 2026-11-04T05:57:15.528965Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-04T05:57:15.529099Z info Proxy role ips=[10.132.0.35] type=router id=maas-default-gateway-openshift-default-fcb75c45f-bm525.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-04T05:57:15.529176Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-04T05:57:15.530603Z info cpu limit detected as 2, setting concurrency 2026-11-04T05:57:15.530794Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-04T05:57:15.530805Z info JWT policy is third-party-jwt 2026-11-04T05:57:15.530808Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-04T05:57:15.532218Z info Starting default Istio SDS Server 2026-11-04T05:57:15.532239Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-04T05:57:15.532256Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-04T05:57:15.532653Z info Opening status port 15020 2026-11-04T05:57:15.533366Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-04T05:57:15.533855Z info sds Starting SDS grpc server 2026-11-04T05:57:15.533868Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-04T05:57:15.534764Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-04T05:57:15.535745Z info Starting proxy agent 2026-11-04T05:57:15.535830Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-04T05:57:15.602216Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-04T05:57:15.604785Z info cache generated new workload certificate resourceName=default latency=70.834878ms ttl=23h59m59.395229041s 2026-11-04T05:57:15.604828Z info cache Root cert has changed, start rotating root cert 2026-11-04T05:57:15.604873Z info cache returned workload trust anchor from cache ttl=23h59m59.395136263s 2026-11-04T05:57:15.744739Z info ads ADS: new connection for node:1 2026-11-04T05:57:15.744796Z info cache returned workload certificate from cache ttl=23h59m59.255206238s 2026-11-04T05:57:15.745035Z info ads ADS: new connection for node:2 2026-11-04T05:57:15.745103Z info cache returned workload trust anchor from cache ttl=23h59m59.254898187s 2026-11-04T05:57:15.832761Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T05:57:24.166425Z info Readiness succeeded in 8.639906913s 2026-11-04T05:57:24.166707Z info Envoy proxy is ready [2026-11-04T05:57:33.945Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 72 - "34.228.250.194,10.133.0.22" "curl/7.76.1" "716f3459-a0ff-44cd-b000-00652a5842a0" "maas.apps.243f2a199f0fd723b411.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.35:443 10.133.0.22:42130 - v1-model-routing [2026-11-04T05:57:34.089Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 68 - "34.228.250.194,10.133.0.22" "curl/7.76.1" "e21cd2c5-bcd5-4f6a-8b06-983e8352e3f3" "maas.apps.243f2a199f0fd723b411.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.35:443 10.133.0.22:42140 - v1-model-routing 2026-11-04T06:00:11.782932Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T06:00:11.782965Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T06:00:11.782982Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=27 2026-11-04T06:00:11.782987Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 [2026-11-04T06:00:11.707Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 75 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "fb26d27d-f9ff-438d-bfd6-8dca53bdaaa3" "maas.apps.243f2a199f0fd723b411.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.35:40352 10.132.0.35:443 10.133.0.22:36440 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T06:00:12.966027Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T06:00:12.966065Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T06:00:12.966082Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T06:00:12.966089Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T06:00:13.036585Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T06:00:13.036620Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T06:00:13.036633Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T06:00:13.036638Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=27 [2026-11-04T06:00:12.906Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 60 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "bff4b78e-49fc-4872-9f41-6be082069213" "maas.apps.243f2a199f0fd723b411.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.35:40366 10.132.0.35:443 10.133.0.22:36450 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T06:00:12.977Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 59 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "77533114-66da-436a-9cab-dd663b0976b1" "maas.apps.243f2a199f0fd723b411.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.35:40352 10.132.0.35:443 10.133.0.22:36456 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T06:00:13.392770Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T06:00:13.392799Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T06:00:13.392811Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=27 2026-11-04T06:00:13.392816Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 [2026-11-04T06:00:13.320Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 71 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "0b22e7c9-f659-40b5-b5d7-9862c8f3a6fe" "maas.apps.243f2a199f0fd723b411.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.35:40352 10.132.0.35:443 10.133.0.22:36466 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T06:00:19.727Z] "POST /maas-api/internal/v1/api-keys/cleanup HTTP/1.1" 404 NR route_not_found - "-" 2 0 13 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "cde637f1-31b1-449e-a50f-8d701d20fa05" "maas.apps.243f2a199f0fd723b411.prod.konflux-ocp-ci.dev" "-" - - 10.132.0.35:443 10.133.0.22:55918 - - 2026-11-04T06:00:22.073818Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=27 [2026-11-04T06:00:21.997Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 278 78 - "34.228.250.194,10.133.0.22" "-" "1a5fbddd-b1b7-4a99-846c-a7b40b16e675" "maas.apps.243f2a199f0fd723b411.prod.konflux-ocp-ci.dev" "10.133.0.51:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.35:56830 10.132.0.35:443 10.133.0.22:55926 - v1-completions-path [2026-11-04T06:00:22.580Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 292 65 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "176114c1-5a7d-4dc6-892f-91babb075ac9" "maas.apps.243f2a199f0fd723b411.prod.konflux-ocp-ci.dev" "10.133.0.51:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.35:56830 10.132.0.35:443 10.133.0.22:55934 - v1-completions-path 2026-11-04T06:01:08.421722Z info ads ADS: new connection for node:3 2026-11-04T06:01:08.424971Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-04T06:01:08.425012Z info cache read certificate from file resource=file-root:system 2026-11-04T06:01:09.026681Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T06:01:23.814721Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T06:01:28.769963Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T06:01:28.769997Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T06:01:28.770011Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T06:01:28.770016Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T06:01:28.699Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 66 309 70 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "6316e16a-d740-42b1-bc03-268cfc9ef4b7" "maas.apps.243f2a199f0fd723b411.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.35:39818 10.132.0.35:443 10.133.0.22:54404 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T06:02:08.557210Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T06:02:54.617706Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T06:03:12.763920Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T06:03:26.900953Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T06:04:29.934670Z info ads ADS: "" 3 terminated with stream closed