2026-11-01T23:18:11.821228Z info FLAG: --concurrency="0" 2026-11-01T23:18:11.821245Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-01T23:18:11.821249Z info FLAG: --help="false" 2026-11-01T23:18:11.821251Z info FLAG: --log_as_json="false" 2026-11-01T23:18:11.821253Z info FLAG: --log_caller="" 2026-11-01T23:18:11.821255Z info FLAG: --log_output_level="default:info" 2026-11-01T23:18:11.821257Z info FLAG: --log_stacktrace_level="default:none" 2026-11-01T23:18:11.821262Z info FLAG: --log_target="[stdout]" 2026-11-01T23:18:11.821264Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-01T23:18:11.821266Z info FLAG: --outlierLogPath="" 2026-11-01T23:18:11.821268Z info FLAG: --profiling="true" 2026-11-01T23:18:11.821270Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-01T23:18:11.821272Z info FLAG: --proxyLogLevel="warning" 2026-11-01T23:18:11.821274Z info FLAG: --serviceCluster="istio-proxy" 2026-11-01T23:18:11.821276Z info FLAG: --stsPort="0" 2026-11-01T23:18:11.821278Z info FLAG: --templateFile="" 2026-11-01T23:18:11.821280Z info FLAG: --tokenManagerPlugin="" 2026-11-01T23:18:11.821283Z info FLAG: --vklog="0" 2026-11-01T23:18:11.821286Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-01T23:18:11.821416Z info Proxy role ips=[10.133.0.248] type=router id=maas-default-gateway-openshift-default-7f694df8bc-4txmj.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-01T23:18:11.821459Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-01T23:18:11.822484Z info cpu limit detected as 2, setting concurrency 2026-11-01T23:18:11.823142Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-01T23:18:11.823154Z info JWT policy is third-party-jwt 2026-11-01T23:18:11.823158Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-01T23:18:11.824521Z info Starting default Istio SDS Server 2026-11-01T23:18:11.824539Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-01T23:18:11.824556Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-01T23:18:11.824624Z info Opening status port 15020 2026-11-01T23:18:11.825540Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-01T23:18:11.827390Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-01T23:18:11.828669Z info sds Starting SDS grpc server 2026-11-01T23:18:11.829094Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-01T23:18:11.829255Z info Starting proxy agent 2026-11-01T23:18:11.829285Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-01T23:18:11.892583Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-01T23:18:11.929413Z info cache generated new workload certificate resourceName=default latency=103.525419ms ttl=23h59m59.070589517s 2026-11-01T23:18:11.929445Z info cache Root cert has changed, start rotating root cert 2026-11-01T23:18:11.929476Z info cache returned workload trust anchor from cache ttl=23h59m59.070524547s 2026-11-01T23:18:12.039858Z info ads ADS: new connection for node:1 2026-11-01T23:18:12.039935Z info cache returned workload certificate from cache ttl=23h59m58.960067643s 2026-11-01T23:18:12.040206Z info ads ADS: new connection for node:2 2026-11-01T23:18:12.040286Z info cache returned workload trust anchor from cache ttl=23h59m58.959716103s 2026-11-01T23:18:12.125606Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:18:20.186305Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:18:21.319124Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-01T23:18:22.137490Z info Readiness succeeded in 10.318846369s 2026-11-01T23:18:22.137803Z info Envoy proxy is ready 2026-11-01T23:18:25.412476Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:18:31.581163Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-01T23:18:44.790Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 89 - "52.71.36.72,10.134.0.21" "python-requests/2.34.2" "189fff96-e786-4bee-9403-34d13b9b9456" "maas.apps.02a4ba9dcb097e09f654.prod.konflux-ocp-ci.dev" "10.132.0.27:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.248:33676 10.133.0.248:443 10.134.0.21:52632 - v1-completions-path [2026-11-01T23:18:46.693Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 58 - "52.71.36.72,10.134.0.21" "python-requests/2.34.2" "60e336ee-8a75-4a9f-af4f-a291045487e5" "maas.apps.02a4ba9dcb097e09f654.prod.konflux-ocp-ci.dev" "10.132.0.27:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.248:33676 10.133.0.248:443 10.134.0.21:52660 - v1-completions-path 2026-11-01T23:19:00.418487Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:19:05.015808Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:19:28.509681Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:19:34.579007Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:20:00.775325Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:20:22.886062Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:20:57.320492Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:20:57.320526Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:20:57.320538Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:20:57.320543Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-01T23:20:56.623Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 75 - "52.71.36.72,10.134.0.21" "python-requests/2.34.2" "8acedc41-c886-4bb9-b25a-83e74567b8ec" "maas.apps.02a4ba9dcb097e09f654.prod.konflux-ocp-ci.dev" "10.134.0.31:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.248:53202 10.133.0.248:443 10.134.0.21:57104 - v1-completions-path [2026-11-01T23:20:57.210Z] "DELETE /v1/api-keys/c22758fb-10bc-4764-a4b1-8874c2080700 HTTP/1.1" 404 - via_upstream - "-" 0 29 110 - "52.71.36.72,10.134.0.21" "python-requests/2.34.2" "628d9f9a-60ae-456f-b85f-766c71e246ab" "maas.apps.02a4ba9dcb097e09f654.prod.konflux-ocp-ci.dev" "10.132.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.248:57154 10.133.0.248:443 10.134.0.21:57114 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T23:21:05.556676Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:21:26.981340Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:21:37.693458Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:24:00.266522Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-01T23:24:00.372777Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:24:00.372809Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:24:00.372823Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:24:00.372829Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-01T23:24:00.257Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 80 322 115 - "52.71.36.72,10.134.0.21" "python-requests/2.34.2" "6a24e647-688e-4feb-9997-f4d45b6bc426" "maas.apps.02a4ba9dcb097e09f654.prod.konflux-ocp-ci.dev" "10.132.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.248:35454 10.133.0.248:443 10.134.0.21:33020 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T23:25:22.538944Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T23:25:22.538988Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T23:25:22.539003Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T23:25:22.539008Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-01T23:25:22.408Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 130 - "52.71.36.72,10.134.0.21" "python-requests/2.34.2" "6dd43613-3b81-41a3-84e3-9d8fbab097f2" "maas.apps.02a4ba9dcb097e09f654.prod.konflux-ocp-ci.dev" "10.132.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.248:41418 10.133.0.248:443 10.134.0.21:60492 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T23:25:44.919825Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:26:01.869374Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:26:06.314537Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:26:16.899988Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:26:16.900025Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:26:16.900040Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-01T23:26:16.900045Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-01T23:26:16.783Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 65 307 116 - "52.71.36.72,10.134.0.21" "python-requests/2.34.2" "683aedb9-0e76-4efc-8d96-688c80345657" "maas.apps.02a4ba9dcb097e09f654.prod.konflux-ocp-ci.dev" "10.132.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.248:33870 10.133.0.248:443 10.134.0.21:42552 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T23:26:31.178042Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:28:27.486135Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:28:27.486183Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:28:27.486199Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:28:27.486204Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-01T23:28:27.370Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 115 - "52.71.36.72,10.134.0.21" "python-requests/2.34.2" "1badbf45-dbe6-4c4d-b04e-525104c6c358" "maas.apps.02a4ba9dcb097e09f654.prod.konflux-ocp-ci.dev" "10.132.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.248:37648 10.133.0.248:443 10.134.0.21:39114 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T23:28:37.486249Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:29:21.110928Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:29:51.112119Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:30:06.293149Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:30:12.573973Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:30:25.653560Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:30:30.304952Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:30:39.031800Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:30:39.031850Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-01T23:30:39.031875Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-01T23:30:39.031883Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-01T23:30:38.962Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 524 69 - "52.71.36.72,10.134.0.21" "python-requests/2.34.2" "1f6afd12-f683-42d9-ba12-3570824781dd" "maas.apps.02a4ba9dcb097e09f654.prod.konflux-ocp-ci.dev" "10.132.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.248:50648 10.133.0.248:443 10.134.0.21:50920 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T23:30:53.698092Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:34:26.027605Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T23:34:26.027629Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T23:34:26.027633Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T23:34:26.027636Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-01T23:34:25.852Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 175 - "52.71.36.72,10.134.0.21" "python-requests/2.34.2" "525a7ac0-e9c0-4d82-bf07-e81326e4f9a6" "maas.apps.02a4ba9dcb097e09f654.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.248:443 10.134.0.21:54284 - v1-catch-all-path 2026-11-01T23:34:29.237408Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:34:44.434414Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:35:05.405215Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:35:09.695474Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:35:25.984072Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-01T23:35:37.669768Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8