2026-10-30T20:06:11.538382Z info FLAG: --concurrency="0" 2026-10-30T20:06:11.538400Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T20:06:11.538406Z info FLAG: --help="false" 2026-10-30T20:06:11.538409Z info FLAG: --log_as_json="false" 2026-10-30T20:06:11.538412Z info FLAG: --log_caller="" 2026-10-30T20:06:11.538415Z info FLAG: --log_output_level="default:info" 2026-10-30T20:06:11.538418Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T20:06:11.538425Z info FLAG: --log_target="[stdout]" 2026-10-30T20:06:11.538428Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T20:06:11.538431Z info FLAG: --outlierLogPath="" 2026-10-30T20:06:11.538434Z info FLAG: --profiling="true" 2026-10-30T20:06:11.538437Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T20:06:11.538441Z info FLAG: --proxyLogLevel="warning" 2026-10-30T20:06:11.538444Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T20:06:11.538447Z info FLAG: --stsPort="0" 2026-10-30T20:06:11.538450Z info FLAG: --templateFile="" 2026-10-30T20:06:11.538453Z info FLAG: --tokenManagerPlugin="" 2026-10-30T20:06:11.538457Z info FLAG: --vklog="0" 2026-10-30T20:06:11.538461Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T20:06:11.538633Z info Proxy role ips=[10.133.0.224] type=router id=maas-default-gateway-openshift-default-6f54779779-d8qbx.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T20:06:11.538692Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T20:06:11.539782Z info cpu limit detected as 2, setting concurrency 2026-10-30T20:06:11.539985Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T20:06:11.539996Z info JWT policy is third-party-jwt 2026-10-30T20:06:11.539999Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T20:06:11.541362Z info Starting default Istio SDS Server 2026-10-30T20:06:11.541389Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T20:06:11.541413Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T20:06:11.541881Z info Opening status port 15020 2026-10-30T20:06:11.542743Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T20:06:11.543423Z info sds Starting SDS grpc server 2026-10-30T20:06:11.543445Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T20:06:11.544015Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T20:06:11.544896Z info Starting proxy agent 2026-10-30T20:06:11.544937Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T20:06:11.615519Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T20:06:11.669349Z info cache generated new workload certificate resourceName=default latency=125.804889ms ttl=23h59m59.330654232s 2026-10-30T20:06:11.669390Z info cache Root cert has changed, start rotating root cert 2026-10-30T20:06:11.669422Z info cache returned workload trust anchor from cache ttl=23h59m59.330578138s 2026-10-30T20:06:11.746497Z info ads ADS: new connection for node:1 2026-10-30T20:06:11.746558Z info cache returned workload certificate from cache ttl=23h59m59.253444207s 2026-10-30T20:06:11.746809Z info ads ADS: new connection for node:2 2026-10-30T20:06:11.746907Z info cache returned workload trust anchor from cache ttl=23h59m59.253095814s 2026-10-30T20:06:11.830963Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:06:19.867413Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:06:20.795988Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-10-30T20:06:21.138333Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-10-30T20:06:21.825535Z info Readiness succeeded in 10.29039687s 2026-10-30T20:06:21.826075Z info Envoy proxy is ready 2026-10-30T20:06:31.597368Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:06:47.144481Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T20:07:03.636Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 83 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "c6dcaecf-396a-4139-a02a-7f092878d64d" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.224:56544 10.133.0.224:443 10.132.0.13:55570 - v1-completions-path 2026-10-30T20:07:04.515697Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-30T20:07:04.375Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 62 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "b3a08151-6543-424e-b23c-14468fe48c1f" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.224:56544 10.133.0.224:443 10.132.0.13:55138 - v1-completions-path [2026-10-30T20:07:04.456Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 62 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "0cc0263c-9f3e-4c3c-bfd2-c9a24aa9974c" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.224:56544 10.133.0.224:443 10.132.0.13:55152 - v1-completions-path [2026-10-30T20:07:05.573Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 69 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "faa61fe0-00c2-43e6-8a85-7035a1a21073" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.224:56560 10.133.0.224:443 10.132.0.13:55180 - v1-completions-path 2026-10-30T20:07:20.753894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:07:31.158104Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:07:35.253380Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T20:07:41.042Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 60 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "3e518c98-a099-4efa-b7c8-1f8e3b38f2db" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.224:56560 10.133.0.224:443 10.132.0.13:37372 - v1-completions-path [2026-10-30T20:07:41.364Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 55 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "f0f4dab7-4c05-4916-b703-b587d4823736" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.224:56544 10.133.0.224:443 10.132.0.13:37380 - v1-completions-path [2026-10-30T20:07:41.763Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 57 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "c5e7753f-9d31-4fb3-b266-b1043db79654" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.224:56560 10.133.0.224:443 10.132.0.13:37396 - v1-completions-path 2026-10-30T20:07:53.726134Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:07:55.987165Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:08:15.187894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:08:19.809260Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:08:36.336234Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:08:38.477972Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:08:57.465446Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:09:08.295553Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:09:11.161857Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:09:30.416121Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:09:32.996465Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:10:04.382341Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:10:06.985281Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:11:54.620719Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:11:57.835532Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:13:27.136873Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:13:42.199841Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:13:44.203832Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:13:47.123362Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-30T20:13:47.368836Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T20:13:47.368857Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T20:13:47.368860Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T20:13:47.368863Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T20:13:47.068Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 74 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "ba5dad3a-3a3b-46c0-b905-b5b6152043ce" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.224:46208 10.133.0.224:443 10.132.0.13:40044 - v1-completions-path [2026-10-30T20:13:47.338Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 30 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "d8ce9526-33fd-4bf2-9eb2-94ad56de2dc4" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.224:443 10.132.0.13:40060 - v1-completions-path 2026-10-30T20:13:59.114780Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:16:12.071918Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T20:16:12.071959Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T20:16:12.071976Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T20:16:12.071981Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T20:16:11.996Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 568 75 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "13998656-87e6-4809-ae29-da76530f7ab9" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.44:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.224:46194 10.133.0.224:443 10.132.0.13:50836 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T20:16:31.410964Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:17:00.253770Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:17:36.158846Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:17:47.241313Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:17:58.371987Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:18:03.384816Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:18:06.498814Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:18:17.215507Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:18:21.106469Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T20:18:25.401Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 62 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "57071b51-4e3a-4a5f-81da-587c19b134d9" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.132.0.45:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.224:56560 10.133.0.224:443 10.132.0.13:44458 - v1-completions-path 2026-10-30T20:18:37.500623Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:19:17.401762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:19:20.389056Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-10-30T20:19:20.311Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 77 - "34.228.250.194,10.132.0.13" "python-requests/2.34.2" "0b4fbcaa-1343-47f6-8654-f5ed8d951331" "maas.apps.88a23b71f093755d6258.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.224:55886 10.133.0.224:443 10.132.0.13:56886 - v1-catch-all-path 2026-10-30T20:19:22.632093Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:19:37.847281Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:19:39.878543Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:19:54.424547Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T20:20:05.310983Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8