2026-11-04T01:55:32.006730Z info FLAG: --concurrency="0" 2026-11-04T01:55:32.006748Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-04T01:55:32.006752Z info FLAG: --help="false" 2026-11-04T01:55:32.006755Z info FLAG: --log_as_json="false" 2026-11-04T01:55:32.006757Z info FLAG: --log_caller="" 2026-11-04T01:55:32.006759Z info FLAG: --log_output_level="default:info" 2026-11-04T01:55:32.006761Z info FLAG: --log_stacktrace_level="default:none" 2026-11-04T01:55:32.006765Z info FLAG: --log_target="[stdout]" 2026-11-04T01:55:32.006767Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-04T01:55:32.006769Z info FLAG: --outlierLogPath="" 2026-11-04T01:55:32.006771Z info FLAG: --profiling="true" 2026-11-04T01:55:32.006773Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-04T01:55:32.006775Z info FLAG: --proxyLogLevel="warning" 2026-11-04T01:55:32.006777Z info FLAG: --serviceCluster="istio-proxy" 2026-11-04T01:55:32.006779Z info FLAG: --stsPort="0" 2026-11-04T01:55:32.006781Z info FLAG: --templateFile="" 2026-11-04T01:55:32.006783Z info FLAG: --tokenManagerPlugin="" 2026-11-04T01:55:32.006786Z info FLAG: --vklog="0" 2026-11-04T01:55:32.006788Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-04T01:55:32.006959Z info Proxy role ips=[10.133.0.35] type=router id=maas-default-gateway-openshift-default-58847f9fc5-p77k5.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-04T01:55:32.007008Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-04T01:55:32.008037Z info cpu limit detected as 2, setting concurrency 2026-11-04T01:55:32.008228Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-04T01:55:32.008238Z info JWT policy is third-party-jwt 2026-11-04T01:55:32.008241Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-04T01:55:32.009945Z info Starting default Istio SDS Server 2026-11-04T01:55:32.009964Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-04T01:55:32.009981Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-04T01:55:32.010111Z info Opening status port 15020 2026-11-04T01:55:32.011270Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-04T01:55:32.012075Z info sds Starting SDS grpc server 2026-11-04T01:55:32.012091Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-04T01:55:32.012735Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-04T01:55:32.013592Z info Starting proxy agent 2026-11-04T01:55:32.013631Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-04T01:55:32.077617Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-04T01:55:32.185267Z info ads ADS: new connection for node:1 2026-11-04T01:55:32.185818Z info ads ADS: new connection for node:2 2026-11-04T01:55:32.210054Z info cache generated new workload certificate resourceName=default latency=197.837392ms ttl=23h59m59.789949089s 2026-11-04T01:55:32.210086Z info cache Root cert has changed, start rotating root cert 2026-11-04T01:55:32.210115Z info cache returned workload certificate from cache ttl=23h59m59.789888167s 2026-11-04T01:55:32.210125Z info cache returned workload trust anchor from cache ttl=23h59m59.789875621s 2026-11-04T01:55:32.210210Z info cache returned workload trust anchor from cache ttl=23h59m59.789791764s 2026-11-04T01:55:32.210508Z info cache returned workload trust anchor from cache ttl=23h59m59.789492962s 2026-11-04T01:55:32.295687Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:55:40.832068Z info Readiness succeeded in 8.827465008s 2026-11-04T01:55:40.832544Z info Envoy proxy is ready 2026-11-04T01:58:03.836669Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:03.836706Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:03.836718Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:58:03.836723Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-04T01:58:03.745Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 84 327 91 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "3c8fb267-bb43-483d-8e0d-d40dbd4fb604" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.35:47872 10.133.0.35:443 10.132.0.15:50800 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:58:04.308637Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:58:04.308676Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:58:04.308688Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:58:04.308694Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T01:58:04.509701Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:04.509738Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:04.509754Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:58:04.509760Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-04T01:58:04.222Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 86 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "97d02d84-e546-49fb-a938-4ad6bb227aab" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.35:47882 10.133.0.35:443 10.132.0.15:50812 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:58:04.326Z] "DELETE /v1/api-keys/05cd9ef5-c26c-408b-b046-e0b52019f890 HTTP/1.1" 200 - via_upstream - "-" 0 321 183 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "d822e811-356a-423e-90ff-f26dbc1127cd" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.35:47872 10.133.0.35:443 10.132.0.15:50818 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:58:05.327140Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.327179Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.327195Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.327202Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T01:58:05.250Z] "DELETE /v1/api-keys/87d31e59-e39f-4d55-a8a6-88139b8fd2b1 HTTP/1.1" 200 - via_upstream - "-" 0 321 76 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "22858912-d7b1-4bbf-897f-9c72fa286704" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.35:47872 10.133.0.35:443 10.132.0.15:50834 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:58:06.965537Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-04T01:58:06.965562Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-04T01:58:06.965566Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-04T01:58:06.965569Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-04T01:58:06.928Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 77 37 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "a2ebf901-9a71-43e2-acb8-687cb1e57423" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.35:443 10.132.0.15:50848 - v1-completions-path [2026-11-04T01:58:07.373Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 90 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "49e56df8-21bc-4aa7-8d07-f19298d0682d" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.133.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:49746 10.133.0.35:443 10.132.0.15:50850 - v1-completions-path [2026-11-04T01:58:07.866Z] "POST /maas-api/internal/v1/api-keys/cleanup HTTP/1.1" 404 NR route_not_found - "-" 2 0 9 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "78266bba-e975-4d9a-ab6e-4b4550502549" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "-" - - 10.133.0.35:443 10.132.0.15:50854 - - 2026-11-04T01:58:09.926731Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:58:09.926764Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:58:09.926775Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:58:09.926780Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T01:58:10.282670Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:10.282703Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:10.282715Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:10.282720Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T01:58:09.918Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 20 8 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "c48cad08-16cf-4777-b796-694328b82de9" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.35:47882 10.133.0.35:443 10.132.0.15:50874 - odh-ai-gateway-infra.maas-api-route.4 [2026-11-04T01:58:10.213Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 86 328 69 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "3b56fe26-4b0d-4d3f-95d6-72bb6d3f5d2f" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.35:47872 10.133.0.35:443 10.132.0.15:50880 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:58:26.051Z] "GET /v1/tenants HTTP/1.1" 404 NR route_not_found - "-" 0 0 9 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "61c5fbab-0dac-4fc0-b2d3-94a4945f2ae7" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "-" - - 10.133.0.35:443 10.132.0.15:38448 - - 2026-11-04T01:59:12.638721Z info ads ADS: new connection for node:3 2026-11-04T01:59:12.646120Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-04T01:59:12.646177Z info cache read certificate from file resource=file-root:system 2026-11-04T01:59:13.436496Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:59:31.666264Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:59:42.492781Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:59:42.492818Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:59:42.492829Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:59:42.492858Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:59:42.493049Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-04T01:59:42.302Z] "POST /anything/e2e-dft-e168bb-cd2527 HTTP/1.1" 200 - via_upstream - "-" 109 1596 190 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "798a48df-eda4-43d2-a2ef-768333d74fd7" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.35:53934 10.133.0.35:443 10.132.0.15:59316 - llm.external-model-e2e-dft-e168bb-e52040f500bf.0 2026-11-04T02:00:21.966813Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:01:06.139738Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:01:27.560386Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:01:32.479700Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:02:26.064073Z info ads ADS: "" 3 terminated with stream closed 2026-11-04T02:02:40.587120Z info ads ADS: new connection for node:4 2026-11-04T02:02:40.590722Z info cache read certificate from file resource=file-root:system 2026-11-04T02:02:41.994253Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:02:50.715680Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T02:02:56.476Z] "POST /llm/e2e-dft-8bf962/anything/e2e-dft-8bf962-2016a7 HTTP/1.1" 404 - - "-" 109 0 36 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "a7df1557-9dc9-4242-97c7-44c7602b1f44" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-8bf962.llm.svc.cluster.local - 10.133.0.35:443 10.132.0.15:41824 - llm.external-model-e2e-dft-8bf962-759bba055c7b.2 2026-11-04T02:03:15.441770Z info ads ADS: "" 4 terminated 2026-11-04T02:03:15.489023Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:03:15.803166Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:03:26.952305Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8