2026-10-31T20:16:42.516479Z info FLAG: --concurrency="0" 2026-10-31T20:16:42.516503Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-31T20:16:42.516509Z info FLAG: --help="false" 2026-10-31T20:16:42.516512Z info FLAG: --log_as_json="false" 2026-10-31T20:16:42.516515Z info FLAG: --log_caller="" 2026-10-31T20:16:42.516518Z info FLAG: --log_output_level="default:info" 2026-10-31T20:16:42.516520Z info FLAG: --log_stacktrace_level="default:none" 2026-10-31T20:16:42.516527Z info FLAG: --log_target="[stdout]" 2026-10-31T20:16:42.516531Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-31T20:16:42.516534Z info FLAG: --outlierLogPath="" 2026-10-31T20:16:42.516537Z info FLAG: --profiling="true" 2026-10-31T20:16:42.516540Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-31T20:16:42.516543Z info FLAG: --proxyLogLevel="warning" 2026-10-31T20:16:42.516546Z info FLAG: --serviceCluster="istio-proxy" 2026-10-31T20:16:42.516549Z info FLAG: --stsPort="0" 2026-10-31T20:16:42.516552Z info FLAG: --templateFile="" 2026-10-31T20:16:42.516554Z info FLAG: --tokenManagerPlugin="" 2026-10-31T20:16:42.516558Z info FLAG: --vklog="0" 2026-10-31T20:16:42.516561Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-31T20:16:42.516762Z info Proxy role ips=[10.134.0.147] type=router id=maas-default-gateway-openshift-default-58d898bd55-6qfjx.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-31T20:16:42.516823Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-31T20:16:42.518009Z info cpu limit detected as 2, setting concurrency 2026-10-31T20:16:42.518270Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-31T20:16:42.518284Z info JWT policy is third-party-jwt 2026-10-31T20:16:42.518288Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-31T20:16:42.520326Z info Starting default Istio SDS Server 2026-10-31T20:16:42.520358Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-31T20:16:42.520395Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-31T20:16:42.520746Z info Opening status port 15020 2026-10-31T20:16:42.521925Z info sds Starting SDS grpc server 2026-10-31T20:16:42.523181Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-31T20:16:42.523162Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-31T20:16:42.525253Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-31T20:16:42.526583Z info Starting proxy agent 2026-10-31T20:16:42.526626Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-31T20:16:42.605770Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-31T20:16:42.658080Z info cache generated new workload certificate resourceName=default latency=134.737241ms ttl=23h59m59.341926575s 2026-10-31T20:16:42.658138Z info cache Root cert has changed, start rotating root cert 2026-10-31T20:16:42.658194Z info cache returned workload trust anchor from cache ttl=23h59m59.34180726s 2026-10-31T20:16:42.766939Z info ads ADS: new connection for node:1 2026-10-31T20:16:42.767041Z info cache returned workload certificate from cache ttl=23h59m59.232960983s 2026-10-31T20:16:42.767694Z info ads ADS: new connection for node:2 2026-10-31T20:16:42.767792Z info cache returned workload trust anchor from cache ttl=23h59m59.232210121s 2026-10-31T20:16:42.857747Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:16:50.955494Z info Readiness succeeded in 8.442423006s 2026-10-31T20:16:50.955850Z info Envoy proxy is ready 2026-10-31T20:17:07.394660Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:17:24.045135Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-31T20:17:37.974Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 91 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "08c7d1d5-1d81-441b-bc11-928b9bdbce71" "maas.apps.9599f41b115b117d03ed.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.147:59730 10.134.0.147:443 10.132.0.14:37326 - v1-completions-path 2026-10-31T20:17:54.916349Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:18:05.554219Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:18:11.534130Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-31T20:18:26.616Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 62 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "831ee11d-91d1-47e8-9749-2f62499b7eac" "maas.apps.9599f41b115b117d03ed.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.147:40472 10.134.0.147:443 10.132.0.14:59794 - v1-completions-path 2026-10-31T20:18:40.450716Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:18:44.822644Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:19:08.514788Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:19:14.549307Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:19:38.759068Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:19:43.154651Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:20:05.612225Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:20:18.710137Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:20:43.033740Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T20:20:43.033775Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T20:20:43.033787Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-31T20:20:43.033793Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-10-31T20:20:42.920Z] "DELETE /v1/api-keys/6f902b95-6797-46be-8456-43dc9cb038ce HTTP/1.1" 404 - via_upstream - "-" 0 29 113 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "e714d30d-1b04-4291-a52e-0091ac27d214" "maas.apps.9599f41b115b117d03ed.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.147:53256 10.134.0.147:443 10.132.0.14:55366 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T20:20:51.132899Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:21:12.424650Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:21:23.251701Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:25:06.704130Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T20:25:06.704157Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T20:25:06.704160Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T20:25:06.704164Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-31T20:25:06.652Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 403 - - "-" 73 113 51 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "1c190c9c-5b32-4a92-9569-894eb54afeec" "maas.apps.9599f41b115b117d03ed.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.147:443 10.132.0.14:52220 - v1-completions-path 2026-10-31T20:25:27.783628Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:25:31.530154Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T20:25:31.530187Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T20:25:31.530199Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T20:25:31.530204Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-31T20:25:31.413Z] "DELETE /v1/api-keys/e2e-trlp-test-key HTTP/1.1" 404 - via_upstream - "-" 0 29 116 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "b60a5098-b0a7-4071-8cd9-19df933be574" "maas.apps.9599f41b115b117d03ed.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.147:39688 10.134.0.147:443 10.132.0.14:41510 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T20:25:44.895277Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:25:49.194952Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:26:13.836250Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:28:21.795342Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:29:04.679939Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:29:35.296990Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:29:50.652580Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:29:58.374228Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:30:10.856581Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:30:15.395227Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:30:21.625629Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T20:30:21.625666Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T20:30:21.625681Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T20:30:21.625687Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-31T20:30:21.501Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 93 335 123 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "261aa81e-c02d-48ef-b722-24efabcda05c" "maas.apps.9599f41b115b117d03ed.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.147:60260 10.134.0.147:443 10.132.0.14:36174 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T20:30:22.305Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 64 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "8500c3a5-8697-4b00-ba9a-430f11b31716" "maas.apps.9599f41b115b117d03ed.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.147:59730 10.134.0.147:443 10.132.0.14:36184 - v1-completions-path 2026-10-31T20:30:36.537122Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:34:12.329911Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:34:24.579772Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:34:31.241624Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 2026-10-31T20:34:31.257667Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=26 [2026-10-31T20:34:31.180Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 77 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "ec03008f-67a7-4b43-9ffa-91d06ec6266b" "maas.apps.9599f41b115b117d03ed.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.147:36114 10.134.0.147:443 10.132.0.14:45342 - v1-catch-all-path 2026-10-31T20:34:31.663024Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=26 [2026-10-31T20:34:31.601Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 61 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "1f62a4cb-6094-4369-aa02-26dccebb8352" "maas.apps.9599f41b115b117d03ed.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.147:36114 10.134.0.147:443 10.132.0.14:45352 - v1-catch-all-path 2026-10-31T20:34:45.361960Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:34:49.692466Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:35:06.054691Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T20:35:17.739094Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9