2026-10-31T00:15:46.459998Z info FLAG: --concurrency="0" 2026-10-31T00:15:46.460016Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-31T00:15:46.460020Z info FLAG: --help="false" 2026-10-31T00:15:46.460022Z info FLAG: --log_as_json="false" 2026-10-31T00:15:46.460024Z info FLAG: --log_caller="" 2026-10-31T00:15:46.460026Z info FLAG: --log_output_level="default:info" 2026-10-31T00:15:46.460028Z info FLAG: --log_stacktrace_level="default:none" 2026-10-31T00:15:46.460032Z info FLAG: --log_target="[stdout]" 2026-10-31T00:15:46.460034Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-31T00:15:46.460036Z info FLAG: --outlierLogPath="" 2026-10-31T00:15:46.460039Z info FLAG: --profiling="true" 2026-10-31T00:15:46.460040Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-31T00:15:46.460042Z info FLAG: --proxyLogLevel="warning" 2026-10-31T00:15:46.460045Z info FLAG: --serviceCluster="istio-proxy" 2026-10-31T00:15:46.460047Z info FLAG: --stsPort="0" 2026-10-31T00:15:46.460048Z info FLAG: --templateFile="" 2026-10-31T00:15:46.460050Z info FLAG: --tokenManagerPlugin="" 2026-10-31T00:15:46.460052Z info FLAG: --vklog="0" 2026-10-31T00:15:46.460055Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-31T00:15:46.460224Z info Proxy role ips=[10.132.0.36] type=router id=maas-default-gateway-openshift-default-58c6585447-7ljch.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-31T00:15:46.460268Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-31T00:15:46.461322Z info cpu limit detected as 2, setting concurrency 2026-10-31T00:15:46.461561Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-31T00:15:46.461572Z info JWT policy is third-party-jwt 2026-10-31T00:15:46.461575Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-31T00:15:46.462741Z info Starting default Istio SDS Server 2026-10-31T00:15:46.462762Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-31T00:15:46.462778Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-31T00:15:46.463120Z info Opening status port 15020 2026-10-31T00:15:46.463995Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-31T00:15:46.465592Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-31T00:15:46.467238Z info sds Starting SDS grpc server 2026-10-31T00:15:46.467256Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-31T00:15:46.467443Z info Starting proxy agent 2026-10-31T00:15:46.467470Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-31T00:15:46.530366Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-31T00:15:46.602272Z info cache generated new workload certificate resourceName=default latency=138.422772ms ttl=23h59m59.397734591s 2026-10-31T00:15:46.602313Z info cache Root cert has changed, start rotating root cert 2026-10-31T00:15:46.602351Z info cache returned workload trust anchor from cache ttl=23h59m59.39765097s 2026-10-31T00:15:46.678656Z info ads ADS: new connection for node:1 2026-10-31T00:15:46.678717Z info cache returned workload certificate from cache ttl=23h59m59.321284877s 2026-10-31T00:15:46.679249Z info ads ADS: new connection for node:2 2026-10-31T00:15:46.679318Z info cache returned workload trust anchor from cache ttl=23h59m59.320682742s 2026-10-31T00:15:46.756715Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:15:54.733285Z info Readiness succeeded in 8.275865666s 2026-10-31T00:15:54.733679Z info Envoy proxy is ready 2026-10-31T00:15:59.130631Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:15:59.130667Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:15:59.130681Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:15:59.130685Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T00:15:58.915Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 55 320 214 - "34.228.250.194,10.133.0.22" "curl/7.76.1" "6e5d415f-5102-45be-b803-765206b969cb" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:44696 10.132.0.36:443 10.133.0.22:41898 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T00:16:00.087Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 440 101 - "34.228.250.194,10.133.0.22" "curl/7.76.1" "5cf5aa2d-a3a1-454a-a341-02f90206c7d2" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.36:54496 10.132.0.36:443 10.133.0.22:41900 - v1-model-routing [2026-10-31T00:16:00.393Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 538 87 - "34.228.250.194,10.133.0.22" "curl/7.76.1" "03a6ed14-d4e5-4aeb-bd19-74b728e1f5b1" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.36:54510 10.132.0.36:443 10.133.0.22:41914 - v1-model-routing 2026-10-31T00:21:49.695897Z info ads ADS: new connection for node:3 2026-10-31T00:21:49.699394Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-31T00:21:49.699440Z info cache read certificate from file resource=file-root:system 2026-10-31T00:21:54.429156Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:21:56.812737Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:22:12.908544Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:22:19.567939Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:22:19.567985Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:22:19.568001Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:22:19.568007Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T00:22:19.568271Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T00:22:19.216Z] "POST /llm/e2e-rt-sw-beb27d/anything/e2e-rt-sw-beb27d-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 352 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "b9a1184a-3323-4bee-b61a-40975e387be4" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-beb27d.llm.svc.cluster.local - 10.132.0.36:443 10.133.0.22:35820 - llm.external-model-e2e-rt-sw-beb27d-f9ca6461eebc.2 2026-10-31T00:22:25.175569Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:22:25.175610Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:22:25.175624Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:22:25.175630Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T00:22:25.175864Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T00:22:25.111Z] "POST /llm/e2e-rt-sw-beb27d/anything/e2e-rt-sw-beb27d-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 64 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "42169262-1599-4e6e-a293-30b5093e0545" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-beb27d.llm.svc.cluster.local - 10.132.0.36:443 10.133.0.22:58118 - llm.external-model-e2e-rt-sw-beb27d-f9ca6461eebc.2 2026-10-31T00:22:27.000997Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:22:46.801290Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:22:46.801325Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:22:46.801337Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:22:46.801343Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T00:22:46.801559Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-31T00:22:46.670Z] "POST /anything/e2e-rt-sw-beb27d-client HTTP/1.1" 200 - via_upstream - "-" 124 1768 131 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "d8dbbea3-658c-4fbd-8a2a-49124691260b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.36:34472 10.132.0.36:443 10.133.0.22:47230 - llm.external-model-e2e-rt-sw-beb27d-f9ca6461eebc.0 2026-10-31T00:22:52.408707Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:22:52.408735Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:22:52.408744Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:22:52.408748Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T00:22:52.408889Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T00:22:52.294Z] "POST /anything/e2e-rt-sw-beb27d-client HTTP/1.1" 200 - via_upstream - "-" 124 1765 114 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "a9c78ae6-c3df-4922-ba8c-b66d0d2dc408" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.36:34476 10.132.0.36:443 10.133.0.22:47244 - llm.external-model-e2e-rt-sw-beb27d-f9ca6461eebc.0 2026-10-31T00:23:33.007724Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:24:02.493901Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:24:02.493934Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:24:02.493944Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:24:02.493949Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T00:24:02.494092Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-31T00:24:02.417Z] "POST /llm/e2e-rt-sw-beb27d/anything/e2e-rt-sw-beb27d-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 76 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "dc2dd684-b1a2-4d03-9069-f76254cae3a8" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-beb27d.llm.svc.cluster.local - 10.132.0.36:443 10.133.0.22:52800 - llm.external-model-e2e-rt-sw-beb27d-f9ca6461eebc.2 2026-10-31T00:24:50.006949Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:25:13.341118Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:26:10.391852Z info ads ADS: "" 3 terminated 2026-10-31T00:26:11.391551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:26:17.419781Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:26:57.832724Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:27:20.485589Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:27:41.075204Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:29:24.842999Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-31T00:30:11.296012Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:11.296048Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:11.296063Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T00:30:11.296068Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-31T00:30:11.177Z] "DELETE /v1/api-keys/c5506a4d-fe75-4ef7-aa8d-dd67db8758a3 HTTP/1.1" 200 - via_upstream - "-" 0 321 118 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "c53d97c2-9e1d-4e90-990f-69fdd788ab19" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:43492 10.132.0.36:443 10.133.0.22:54158 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T00:30:11.502Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 272 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "0de99132-18d1-46b6-9e0f-619950312462" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.132.0.36:443 10.133.0.22:54172 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T00:30:14.985637Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:14.985678Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:14.985698Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T00:30:14.985705Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:14.985715Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-31T00:30:14.869Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 115 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "cafbf01b-9705-494e-b2e0-aa4b1f00d059" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:43492 10.132.0.36:443 10.133.0.22:44252 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T00:30:17.721204Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:17.721257Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:17.721281Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:17.721289Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T00:30:18.092849Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:18.092905Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:18.092929Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:18.092936Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T00:30:17.604Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 117 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "7980ad55-c240-498a-b314-c3998c8dc641" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:43492 10.132.0.36:443 10.133.0.22:44272 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T00:30:17.924Z] "DELETE /v1/api-keys/dd6c8a56-334d-4a4c-80e5-73d3c309d0ac HTTP/1.1" 200 - via_upstream - "-" 0 321 168 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "3eea97d2-6018-4847-8eb7-e94fc9a9653d" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:43492 10.132.0.36:443 10.133.0.22:44274 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T00:30:20.009932Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-31T00:30:20.015206Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:30:20.015237Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:30:20.015253Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T00:30:20.015257Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-31T00:30:19.906Z] "DELETE /v1/api-keys/4934bc49-fc72-4e8e-ae48-2f7e2b2eef40 HTTP/1.1" 200 - via_upstream - "-" 0 321 109 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "10097ad9-9973-4cd4-b5b7-7c3a86759227" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:53442 10.132.0.36:443 10.133.0.22:44280 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T00:30:29.484Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 283 75 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "3ee522d0-9bed-46d5-a6f7-407c4a83156e" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.36:54510 10.132.0.36:443 10.133.0.22:53040 - v1-completions-path 2026-10-31T00:30:36.774845Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T00:30:36.774871Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T00:30:36.774874Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-31T00:30:36.774878Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-31T00:30:36.710Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 77 64 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "129f9d9e-a64e-4a81-ba52-2dda4e0f2b2a" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.36:443 10.133.0.22:55156 - v1-completions-path [2026-10-31T00:30:42.630Z] "POST /maas-api/internal/v1/subscriptions/select HTTP/1.1" 404 NR route_not_found - "-" 56 0 8 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "d3c55d9f-988e-4d6d-8e97-44d3abe57207" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "-" - - 10.132.0.36:443 10.133.0.22:55170 - - 2026-10-31T00:30:43.629252Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:43.629830Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:43.629863Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T00:30:43.629869Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:30:44.107661Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:30:44.107707Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-31T00:30:44.107732Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-31T00:30:44.107740Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-31T00:30:43.620Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 9 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "5c3e30a5-568d-481a-b15a-6aed11e4ec75" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:43492 10.132.0.36:443 10.133.0.22:55188 - odh-ai-gateway-infra.maas-api-route.4 [2026-10-31T00:30:43.994Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 113 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "ac3a628d-0c13-435e-aa8b-29abb632d869" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:53442 10.132.0.36:443 10.133.0.22:55198 - odh-ai-gateway-infra.maas-api-route.3 2026-10-31T00:36:25.259032Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:36:25.259072Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:36:25.259088Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:36:25.259093Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T00:36:25.133Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 69 312 125 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "f44adb92-6d65-4d44-8d3a-a8aa4ec4fdf8" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:56912 10.132.0.36:443 10.133.0.22:42362 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T00:36:25.489Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 110 321 152 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "a20c859c-7fd3-451f-b779-851dbf8a8526" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.36:54510 10.132.0.36:443 10.133.0.22:42366 - v1-chat-completions-path [2026-10-31T00:45:09.215Z] "POST /maas-api/v1/tokens HTTP/1.1" 401 - - "-" 20 88 22 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "25250832-ee99-4525-8637-252152a88fe8" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.132.0.36:443 10.133.0.22:32806 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T00:45:10.106Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 136 346 91 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "be8d86af-e166-404f-b38b-e2f60b04e9f9" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.36:54496 10.132.0.36:443 10.133.0.22:32812 - v1-chat-completions-path [2026-10-31T00:45:10.402Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 135 1812 68 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "70752e39-c2a0-434d-aa9a-70f49efee96f" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.36:54496 10.132.0.36:443 10.133.0.22:32828 - v1-chat-completions-path 2026-10-31T00:45:39.990637Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-31T00:47:48.942439Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:47:48.942485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:47:48.942506Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:47:48.942511Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T00:47:48.822Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 315 119 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "60467dab-fd7b-45f3-a9a4-9cd3f8a3d1fd" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:36576 10.132.0.36:443 10.133.0.22:44512 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-31T00:47:49.147Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 113 316 71 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "6c46a482-f382-420e-840c-5d8e882e3ef9" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.36:54496 10.132.0.36:443 10.133.0.22:44516 - v1-chat-completions-path 2026-10-31T00:47:49.754158Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:47:49.754193Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:47:49.754207Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:47:49.754211Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-31T00:47:49.754360Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 2026-10-31T00:47:49.754422Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-31T00:47:49.754431Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-31T00:47:49.754434Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-31T00:47:49.754437Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-31T00:47:49.424Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 113 316 67 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "25f25923-10da-4fe0-afd7-ef47bcca3470" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.36:54496 10.132.0.36:443 10.133.0.22:44530 - v1-chat-completions-path [2026-10-31T00:47:49.697Z] "POST /v1/chat/completions HTTP/1.1" 404 - via_upstream - "-" 126 129 57 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "25e0efb0-2750-4e2d-b961-3b08e028c88b" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.36:54496 10.132.0.36:443 10.133.0.22:44538 - v1-chat-completions-path 2026-10-31T00:49:41.180906Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:49:41.180939Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:49:41.180952Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-31T00:49:41.180956Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-31T00:49:41.170Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 10 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "cbbba210-78ae-4c80-9e74-bf68ec3977ba" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "10.132.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.36:48038 10.132.0.36:443 10.133.0.22:41204 - odh-ai-gateway-infra.maas-api-route.4 [2026-10-31T00:57:45.161Z] "GET /v1/tenants HTTP/1.1" 404 NR route_not_found - "-" 0 0 8 - "34.228.250.194,10.133.0.22" "python-requests/2.34.2" "21d33d30-47b1-494d-b2f3-bc8b4fdc9580" "maas.apps.8177d3f27445983bd214.prod.konflux-ocp-ci.dev" "-" - - 10.132.0.36:443 10.133.0.22:50952 - - 2026-10-31T01:10:46.487210Z info ads ADS: new connection for node:4 2026-10-31T01:10:46.494117Z info cache read certificate from file resource=file-root:system 2026-10-31T01:13:04.787808Z info ads ADS: "" 4 terminated with stream closed