2026-10-30T12:54:25.981557Z info FLAG: --concurrency="0" 2026-10-30T12:54:25.981573Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T12:54:25.981577Z info FLAG: --help="false" 2026-10-30T12:54:25.981579Z info FLAG: --log_as_json="false" 2026-10-30T12:54:25.981582Z info FLAG: --log_caller="" 2026-10-30T12:54:25.981584Z info FLAG: --log_output_level="default:info" 2026-10-30T12:54:25.981586Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T12:54:25.981604Z info FLAG: --log_target="[stdout]" 2026-10-30T12:54:25.981608Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T12:54:25.981611Z info FLAG: --outlierLogPath="" 2026-10-30T12:54:25.981614Z info FLAG: --profiling="true" 2026-10-30T12:54:25.981618Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T12:54:25.981620Z info FLAG: --proxyLogLevel="warning" 2026-10-30T12:54:25.981623Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T12:54:25.981625Z info FLAG: --stsPort="0" 2026-10-30T12:54:25.981627Z info FLAG: --templateFile="" 2026-10-30T12:54:25.981629Z info FLAG: --tokenManagerPlugin="" 2026-10-30T12:54:25.981632Z info FLAG: --vklog="0" 2026-10-30T12:54:25.981634Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T12:54:25.981796Z info Proxy role ips=[10.132.0.40] type=router id=maas-default-gateway-openshift-default-68cb9b4b95-7mznj.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T12:54:25.981843Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T12:54:25.982873Z info cpu limit detected as 2, setting concurrency 2026-10-30T12:54:25.983429Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T12:54:25.983440Z info JWT policy is third-party-jwt 2026-10-30T12:54:25.983443Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T12:54:25.985169Z info Starting default Istio SDS Server 2026-10-30T12:54:25.985205Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T12:54:25.985229Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T12:54:25.986147Z info Opening status port 15020 2026-10-30T12:54:25.986307Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T12:54:25.987548Z info sds Starting SDS grpc server 2026-10-30T12:54:25.987572Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T12:54:25.987765Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T12:54:25.988635Z info Starting proxy agent 2026-10-30T12:54:25.988665Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T12:54:26.053006Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T12:54:26.181836Z info ads ADS: new connection for node:1 2026-10-30T12:54:26.182880Z info ads ADS: new connection for node:2 2026-10-30T12:54:26.270208Z info cache generated new workload certificate resourceName=default latency=282.436376ms ttl=23h59m59.729797104s 2026-10-30T12:54:26.270240Z info cache Root cert has changed, start rotating root cert 2026-10-30T12:54:26.270287Z info cache returned workload trust anchor from cache ttl=23h59m59.729714324s 2026-10-30T12:54:26.270300Z info cache returned workload certificate from cache ttl=23h59m59.729700182s 2026-10-30T12:54:26.270431Z info cache returned workload trust anchor from cache ttl=23h59m59.729571811s 2026-10-30T12:54:26.270795Z info cache returned workload trust anchor from cache ttl=23h59m59.729206732s 2026-10-30T12:54:26.369521Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T12:54:34.644984Z info Readiness succeeded in 8.666291627s 2026-10-30T12:54:34.645468Z info Envoy proxy is ready 2026-10-30T12:57:34.415413Z info ads ADS: new connection for node:3 2026-10-30T12:57:34.420772Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T12:57:34.420977Z info cache read certificate from file resource=file-root:system 2026-10-30T12:57:35.592620Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T12:57:42.737679Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T12:57:51.754551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T12:58:02.965693Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T12:58:17.444227Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T12:58:17.444265Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T12:58:17.444276Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T12:58:17.444281Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T12:58:17.444464Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T12:58:17.276Z] "POST /anything/e2e-rt-sw-b88fb2-client HTTP/1.1" 200 - via_upstream - "-" 124 1760 168 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "9eee6d68-e339-45e0-9561-18c4b1472ba2" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.40:56522 10.132.0.40:443 10.132.0.13:39308 - llm.external-model-e2e-rt-sw-b88fb2-68098a466cff.0 2026-10-30T12:58:18.643244Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T12:58:18.643275Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T12:58:18.643291Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T12:58:18.643295Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T12:58:18.643451Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T12:58:18.365Z] "POST /llm/e2e-rt-sw-b88fb2/anything/e2e-rt-sw-b88fb2-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 278 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "4640e384-e7d6-4e6b-ab06-07047b7d9810" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-b88fb2.llm.svc.cluster.local - 10.132.0.40:443 10.132.0.13:39324 - llm.external-model-e2e-rt-sw-b88fb2-68098a466cff.2 2026-10-30T12:58:23.870292Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T12:58:34.309605Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T12:58:34.309642Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T12:58:34.309655Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T12:58:34.309661Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T12:58:34.309874Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T12:58:34.133Z] "POST /anything/e2e-rt-sw-b88fb2-client HTTP/1.1" 200 - via_upstream - "-" 124 1761 175 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "94b5de13-7ba9-4d2a-bfa2-f04cc9d4fce3" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.40:33782 10.132.0.40:443 10.132.0.13:33470 - llm.external-model-e2e-rt-sw-b88fb2-68098a466cff.0 2026-10-30T12:58:53.692719Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T12:59:03.921455Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T12:59:09.605756Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T12:59:09.605786Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T12:59:09.605796Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T12:59:09.605801Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T12:59:09.605957Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T12:59:09.451Z] "POST /anything/e2e-rt-eq-f3b60e-client HTTP/1.1" 200 - via_upstream - "-" 124 1759 154 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "93e17a0e-d942-4d2d-b7ca-36184ce4d910" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.40:52334 10.132.0.40:443 10.132.0.13:59704 - llm.external-model-e2e-rt-eq-f3b60e-18b5414f7fac.0 2026-10-30T12:59:36.118754Z info ads ADS: "" 3 terminated with stream closed 2026-10-30T12:59:36.511579Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T12:59:43.221620Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:01:04.422271Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:01:28.305838Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:02:42.383183Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:03:08.403117Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:04:55.490830Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T13:04:55.490864Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T13:04:55.490880Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T13:04:55.490887Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T13:04:55.411Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 79 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "6efd2216-b95c-4ee5-829e-60f371a80b8d" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.40:60094 10.132.0.40:443 10.132.0.13:42946 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T13:04:58.269104Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T13:04:58.269140Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T13:04:58.269154Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T13:04:58.269160Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T13:04:58.198Z] "DELETE /v1/api-keys/9adc6bdb-7df2-46c7-8932-8d3bad40ca7e HTTP/1.1" 200 - via_upstream - "-" 0 321 70 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "42f25618-fcfa-429a-ad6a-82b96257cf29" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.40:60100 10.132.0.40:443 10.132.0.13:42952 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T13:04:58.620929Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T13:04:58.620954Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T13:04:58.620958Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T13:04:58.620961Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-30T13:04:58.585Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 77 35 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "1cb6b449-33ff-4743-b1ce-a393cc1b9617" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.40:443 10.132.0.13:42956 - v1-completions-path 2026-10-30T13:05:01.641679Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T13:05:01.641714Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T13:05:01.641729Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T13:05:01.641734Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T13:05:01.570Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 70 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "560ba6cb-c287-4add-8bfc-ca254eabdb62" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.40:60100 10.132.0.40:443 10.132.0.13:42974 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T13:05:13.559279Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T13:05:13.559323Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T13:05:13.559336Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T13:05:13.559341Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T13:05:13.548Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 10 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "c1d6de61-04d9-4a9b-b948-1e3088411fe0" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.40:60094 10.132.0.40:443 10.132.0.13:52922 - odh-ai-gateway-infra.maas-api-route.4 2026-10-30T13:11:51.747212Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-10-30T13:11:51.790051Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T13:11:51.790092Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T13:11:51.790109Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T13:11:51.790114Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T13:11:51.665Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 315 124 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "8f4df3cf-4464-480b-a879-9908c5f98630" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.40:48636 10.132.0.40:443 10.132.0.13:33030 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T13:20:33.039Z] "GET /v1/tenants HTTP/1.1" 404 NR route_not_found - "-" 0 0 11 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "7e7ad682-3dad-439b-8046-e90bc54d6807" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "-" - - 10.132.0.40:443 10.132.0.13:36906 - - 2026-10-30T13:22:02.193428Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T13:23:28.521855Z info ads ADS: new connection for node:4 2026-10-30T13:23:28.531445Z info cache read certificate from file resource=file-root:system 2026-10-30T13:31:11.493023Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:31:37.561750Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:31:53.013603Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:31:56.346892Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:32:01.774548Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:32:09.127300Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:32:14.346027Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:32:17.759984Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:32:18.329332Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:32:39.245405Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:33:27.564301Z info ads ADS: new connection for node:5 2026-10-30T13:33:27.564473Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T13:33:27.564494Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T13:36:38.695632Z info ads ADS: "" 5 terminated with stream closed 2026-10-30T13:38:22.819819Z info ads ADS: "" 4 terminated with stream closed 2026-10-30T13:43:08.437982Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-10-30T13:44:31.640370Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:45:56.161131Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:49:34.371290Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:50:20.699187Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-10-30T13:51:39.499444Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T13:55:16.471490Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=4 2026-10-30T14:01:10.005325Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:01:12.573555Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T14:01:12.573626Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T14:01:12.573653Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T14:01:12.573664Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T14:01:12.489Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 84 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "2e091b9e-2e39-43dc-b91b-4b8d7b0baf8e" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.40:59494 10.132.0.40:443 10.132.0.13:52950 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T14:01:36.514787Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:02:09.703959Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:02:13.858Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 2817 3089 77 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "ab7ae9ae-f52e-4c9a-b6fb-ba03ea0a8469" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:49500 10.132.0.40:443 10.132.0.13:37920 - v1-chat-completions-path [2026-10-30T14:02:14.137Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 5518 5791 58 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "0c0ef027-756b-4137-9294-437a3ab9c452" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:49500 10.132.0.40:443 10.132.0.13:37934 - v1-chat-completions-path [2026-10-30T14:02:14.593Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 140 405 60 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "46ce129b-824f-4ab4-a076-46b02e751358" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:49500 10.132.0.40:443 10.132.0.13:37938 - v1-chat-completions-path 2026-10-30T14:02:26.774125Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:02:29.330001Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:02:39.807923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:02:45.997Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 429 - - "-" 61 18 58 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "16023602-46b0-4ad3-99ed-549daad27cbf" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.40:443 10.132.0.13:48218 - v1-catch-all-path 2026-10-30T14:02:58.072045Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:03:00.439918Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:03:15.336636Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:03:26.370021Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:03:28.238670Z info ads ADS: new connection for node:6 2026-10-30T14:03:28.238825Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T14:03:28.238853Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T14:03:30.355611Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:04:11.131199Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:04:16.475Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 329 86 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "46c28204-9f7b-4cd1-9998-8f7f30855536" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.1.3:8000" outbound|54321||e2e-prio-pool-6a03d47f-inference-pool-ip-d0654a22.llm.svc.cluster.local 10.132.0.40:54334 10.132.0.40:443 10.132.0.13:55238 - v1-model-routing 2026-10-30T14:04:30.212824Z info ads ADS: "" 6 terminated with stream closed 2026-10-30T14:04:30.257486Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:04:32.706Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 71 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "6a49029d-2f02-4134-a12b-3f2e4716b998" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:53814 10.132.0.40:443 10.132.0.13:41794 - v1-completions-path 2026-10-30T14:04:33.666527Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:04:41.648547Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:06:15.430146Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:06:33.650481Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:06:48.373280Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:07:00.447946Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:07:04.103475Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:07:14.335854Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:07:18.910530Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:07:24.633Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 77 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "627c7044-6417-44ef-8903-5f67beec2dbd" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:58780 10.132.0.40:443 10.132.0.13:45926 - v1-completions-path 2026-10-30T14:07:37.015411Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:08:23.485908Z info ads ADS: new connection for node:7 2026-10-30T14:08:23.491196Z info cache read certificate from file resource=file-root:system 2026-10-30T14:08:24.409268Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:08:26.226886Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:08:42.007706Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-30T14:08:42.058343Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:08:42.058374Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:08:42.058387Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:08:42.058391Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T14:08:42.058544Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T14:08:41.941Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1707 117 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "334ec669-e2a1-4201-bad8-523ec8563899" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.40:44538 10.132.0.40:443 10.132.0.13:43078 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T14:08:52.692762Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T14:08:52.692797Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T14:08:52.692809Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T14:08:52.692814Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T14:08:52.692964Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T14:08:52.589Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1716 103 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "5d83d06a-b2fe-4508-9830-da6cb55c51c9" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.40:44330 10.132.0.40:443 10.132.0.13:44774 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T14:09:07.628802Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:09:07.628833Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:09:07.628843Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T14:09:07.628847Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:09:07.628985Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T14:09:07.531Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1718 97 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "1fc23129-5430-4f6b-b19b-a643f61e0f02" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.40:44538 10.132.0.40:443 10.132.0.13:57478 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T14:09:33.141440Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:09:33.141474Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:09:33.141486Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T14:09:33.141490Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:09:33.141678Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T14:09:33.042Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1717 99 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "44712a3e-3a60-411b-bca3-4abba86f4297" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.40:44538 10.132.0.40:443 10.132.0.13:43978 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T14:09:39.277431Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:09:39.277469Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:09:39.277487Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T14:09:39.277492Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T14:09:39.200Z] "DELETE /v1/api-keys/fa6cc29a-33c2-41d8-86a1-d9e4a62ab6b1 HTTP/1.1" 200 - via_upstream - "-" 0 359 77 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "bff711bd-5fcf-4c6b-9cfa-f8040d5bd928" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.40:34582 10.132.0.40:443 10.132.0.13:38972 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T14:09:50.857302Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:09:50.998202Z info ads ADS: "" 7 terminated with stream closed 2026-10-30T14:09:52.475183Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:09:54.173123Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T14:09:54.173152Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T14:09:54.173155Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T14:09:54.173159Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-30T14:09:54.120Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 52 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "ff0e6460-d4e9-40c2-99d1-6f694e45610d" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.40:443 10.132.0.13:39098 - v1-completions-path 2026-10-30T14:09:59.172811Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:10:16.356692Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:10:21.000292Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:10:24.104733Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:10:24.104775Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:10:24.104796Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:10:24.104803Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T14:10:24.030Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 93 336 74 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "7beedf5b-a4f0-4085-a307-8459624f9bb6" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.40:34582 10.132.0.40:443 10.132.0.13:54072 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T14:10:24.330Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 76 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "141ec12d-f697-40e2-a450-7c7c42490e77" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.40:443 10.132.0.13:54088 - v1-completions-path 2026-10-30T14:10:36.498629Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:10:52.384894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:10:59.138Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 59 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "33e62d86-b160-4804-99d9-c097a3c7dbd5" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:39628 10.132.0.40:443 10.132.0.13:46544 - v1-completions-path 2026-10-30T14:11:16.031971Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:11:26.336963Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:11:31.013653Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:11:35.355Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 58 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "e9049efb-f80f-4bb4-8fb8-bcb13c5a1368" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:39628 10.132.0.40:443 10.132.0.13:60530 - v1-completions-path [2026-10-30T14:11:35.509Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 57 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "04de21d0-4fa4-452d-8c58-21379af0c25b" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:58780 10.132.0.40:443 10.132.0.13:60536 - v1-completions-path 2026-10-30T14:11:47.911747Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:11:50.395110Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:12:09.599505Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:12:14.247431Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:12:29.816391Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:12:32.159288Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:12:50.008194Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:13:00.795860Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:13:03.839000Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:13:10.108054Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:13:10.108100Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:13:10.108122Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T14:13:10.108129Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T14:13:10.032Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 75 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "ba4d47d6-c9c8-4c7d-bf37-a2952ce49ff5" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.40:59238 10.132.0.40:443 10.132.0.13:49322 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T14:13:31.863821Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:13:37.567Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 63 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "3a1bc5d6-8d7c-459a-a069-cdfb8fae027a" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:58780 10.132.0.40:443 10.132.0.13:53486 - v1-completions-path 2026-10-30T14:13:49.975517Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:14:00.192430Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:17:36.170333Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:17:38.337Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 73 295 278 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "e9de9de8-20a9-4558-a457-e0f23a4e5453" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.33:8000" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:34802 10.132.0.40:443 10.132.0.13:34428 - v1-completions-path 2026-10-30T14:17:51.385449Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:17:53.666428Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:18:00.555Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 299 66 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "c0087762-dddc-407c-81e7-0ba3ccd9b4a0" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:53266 10.132.0.40:443 10.132.0.13:47650 - v1-completions-path 2026-10-30T14:18:12.777432Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:18:25.011684Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:18:27.953274Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:18:28.428762Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:18:28.428805Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:18:28.428824Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T14:18:28.428829Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T14:18:28.335Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 315 93 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "0431e929-e6f3-4690-9499-a70bb366cfc7" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.40:56010 10.132.0.40:443 10.132.0.13:45566 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T14:18:43.612386Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:18:54.512487Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:18:59.441236Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:19:16.535785Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:19:21.056561Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:19:24.664958Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:19:31.417Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 61 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "e6569beb-a94a-43ad-97e9-d96fe317f7cd" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:39628 10.132.0.40:443 10.132.0.13:51278 - v1-completions-path [2026-10-30T14:19:32.756Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 55 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "10db0e84-59a4-45fa-9969-b34c51f8cde3" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:58780 10.132.0.40:443 10.132.0.13:51288 - v1-completions-path [2026-10-30T14:19:33.106Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 57 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "c8af2b14-b2d1-44f1-9fcb-36b4bb806c85" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:39628 10.132.0.40:443 10.132.0.13:51296 - v1-completions-path 2026-10-30T14:19:47.368958Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:19:57.740207Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:20:02.352697Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:20:05.976395Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:20:10.641727Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T14:20:15.399Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 66 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "3f485ef4-e664-4c97-8c26-775f3d48d249" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:58780 10.132.0.40:443 10.132.0.13:48786 - v1-completions-path [2026-10-30T14:20:16.692Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 58 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "09276bc6-c6d4-44c9-9cc3-8f56523fb473" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "10.132.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.40:58780 10.132.0.40:443 10.132.0.13:48790 - v1-completions-path 2026-10-30T14:20:29.091765Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:20:39.409498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:20:41.672729Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:20:51.382141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:20:54.974932Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:21:13.345261Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:21:33.349389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:21:43.767252Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:21:54.029987Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:22:04.393867Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:22:14.911736Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:22:25.236476Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:22:35.681420Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:22:46.134187Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:22:56.503546Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:23:06.826044Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T14:23:13.689459Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T14:23:13.689485Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T14:23:13.689488Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T14:23:13.689492Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T14:23:13.662Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 27 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "c066fcd7-aaec-40ec-97a6-c0543e936dc1" "maas.apps.0d483a2e2de579d84691.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.40:443 10.132.0.13:32828 - v1-completions-path