2026-10-30T05:43:18.499267Z info FLAG: --concurrency="0" 2026-10-30T05:43:18.499286Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T05:43:18.499290Z info FLAG: --help="false" 2026-10-30T05:43:18.499292Z info FLAG: --log_as_json="false" 2026-10-30T05:43:18.499294Z info FLAG: --log_caller="" 2026-10-30T05:43:18.499296Z info FLAG: --log_output_level="default:info" 2026-10-30T05:43:18.499298Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T05:43:18.499302Z info FLAG: --log_target="[stdout]" 2026-10-30T05:43:18.499304Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T05:43:18.499308Z info FLAG: --outlierLogPath="" 2026-10-30T05:43:18.499310Z info FLAG: --profiling="true" 2026-10-30T05:43:18.499312Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T05:43:18.499314Z info FLAG: --proxyLogLevel="warning" 2026-10-30T05:43:18.499316Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T05:43:18.499318Z info FLAG: --stsPort="0" 2026-10-30T05:43:18.499320Z info FLAG: --templateFile="" 2026-10-30T05:43:18.499322Z info FLAG: --tokenManagerPlugin="" 2026-10-30T05:43:18.499325Z info FLAG: --vklog="0" 2026-10-30T05:43:18.499327Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T05:43:18.499593Z info Proxy role ips=[10.132.0.60] type=router id=maas-default-gateway-openshift-default-84587756dc-ctspw.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T05:43:18.499642Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T05:43:18.500683Z info cpu limit detected as 2, setting concurrency 2026-10-30T05:43:18.501220Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T05:43:18.501231Z info JWT policy is third-party-jwt 2026-10-30T05:43:18.501234Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T05:43:18.502807Z info Starting default Istio SDS Server 2026-10-30T05:43:18.502831Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T05:43:18.502855Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T05:43:18.503199Z info Opening status port 15020 2026-10-30T05:43:18.504253Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T05:43:18.505573Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T05:43:18.506523Z info Starting proxy agent 2026-10-30T05:43:18.506556Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T05:43:18.515544Z info sds Starting SDS grpc server 2026-10-30T05:43:18.515570Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T05:43:18.570652Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T05:43:18.698343Z info cache generated new workload certificate resourceName=default latency=194.386861ms ttl=23h59m59.301661846s 2026-10-30T05:43:18.698392Z info cache Root cert has changed, start rotating root cert 2026-10-30T05:43:18.698431Z info cache returned workload trust anchor from cache ttl=23h59m59.30156957s 2026-10-30T05:43:18.699445Z info ads ADS: new connection for node:1 2026-10-30T05:43:18.699570Z info cache returned workload certificate from cache ttl=23h59m59.300430924s 2026-10-30T05:43:18.699925Z info ads ADS: new connection for node:2 2026-10-30T05:43:18.699994Z info cache returned workload trust anchor from cache ttl=23h59m59.300007071s 2026-10-30T05:43:18.783280Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:43:26.661397Z info Readiness succeeded in 8.164696089s 2026-10-30T05:43:26.661748Z info Envoy proxy is ready [2026-10-30T05:43:40.571Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 414 82 - "34.228.250.194,10.132.0.20" "curl/7.76.1" "47fb9a8e-0ee0-4421-b43f-117d1a2ce981" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:50460 10.132.0.60:443 10.132.0.20:59458 - v1-model-routing [2026-10-30T05:43:41.451Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 52 - "34.228.250.194,10.132.0.20" "curl/7.76.1" "290ddb58-0b23-45c4-951f-d716bec693a0" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.60:443 10.132.0.20:59474 - v1-model-routing [2026-10-30T05:43:41.624Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 56 - "34.228.250.194,10.132.0.20" "curl/7.76.1" "16399bc4-5de8-498b-b7ce-947190f513bf" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.60:443 10.132.0.20:59476 - v1-model-routing [2026-10-30T05:46:17.766Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 83 0 63 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "4aa8a727-4d0a-4fdf-b4d1-211886748922" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.132.0.60:443 10.132.0.20:33658 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:46:20.399182Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:46:20.399218Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:46:20.399232Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T05:46:20.399236Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T05:46:20.295Z] "DELETE /v1/api-keys/7d538af7-de95-4d69-b1e3-c82512133b2c HTTP/1.1" 200 - via_upstream - "-" 0 321 103 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "06006b33-e65f-4f8f-8ca5-900b2e75705d" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.60:54458 10.132.0.60:443 10.132.0.20:33682 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:46:20.961280Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:46:20.961407Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:46:20.961500Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T05:46:20.961553Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T05:46:20.713Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 84 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "2e79c7e7-4a66-487b-8316-38015f7ca060" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.132.0.60:443 10.132.0.20:33690 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T05:46:20.830Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 130 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "7b87145b-6c94-45b9-825a-e59bd0ccfcb8" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.60:54458 10.132.0.60:443 10.132.0.20:33694 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T05:46:26.736Z] "POST /maas-api/internal/v1/api-keys/cleanup HTTP/1.1" 404 NR route_not_found - "-" 2 0 12 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "11ce03e6-cfc1-412a-928c-6f904f55c755" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "-" - - 10.132.0.60:443 10.132.0.20:49258 - - 2026-10-30T05:46:42.176071Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:46:49.082Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 75 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "2df7f79d-f1b9-453a-96c2-33f0fcfeaa93" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:50460 10.132.0.60:443 10.132.0.20:33568 - v1-completions-path 2026-10-30T05:46:56.681300Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:49:54.831Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 136 382 76 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "f1b1be53-4a89-4066-8a6e-fc9e6fc5beeb" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:50460 10.132.0.60:443 10.132.0.20:32820 - v1-chat-completions-path 2026-10-30T05:52:41.023107Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:53:40.587063Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:54:20.290660Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:56:09.262888Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:16:06.200328Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 [2026-10-30T06:21:02.084Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 110 327 88 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "fe061b10-748f-457a-ae77-b4fc81d14a1b" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:50460 10.132.0.60:443 10.132.0.20:60364 - v1-chat-completions-path 2026-10-30T06:21:05.322711Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:21:05.322756Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:21:05.322771Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:21:05.322778Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T06:21:05.322981Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-10-30T06:21:05.323042Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T06:21:05.323052Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T06:21:05.323056Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T06:21:05.323060Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T06:21:05.255Z] "POST /v1/chat/completions HTTP/1.1" 404 - via_upstream - "-" 120 126 67 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "6c91aa97-8485-4e1b-acc9-1a5ed4748827" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:37220 10.132.0.60:443 10.132.0.20:60370 - v1-chat-completions-path 2026-10-30T06:24:50.705882Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:25:15.932772Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:25:56.792081Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:26:00.956311Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:26:00.956356Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:26:00.956371Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:26:00.956376Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T06:26:00.879Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 87 329 77 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "ee9dca6e-a2b5-4d74-9c6f-fd3db7ec803d" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.60:35760 10.132.0.60:443 10.132.0.20:48922 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:26:01.237504Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-30T06:26:01.181Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 71 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "7cbf7110-a196-4575-8893-754852607455" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:55400 10.132.0.60:443 10.132.0.20:48926 - v1-completions-path 2026-10-30T06:26:13.914089Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:26:17.944336Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:26:21.756Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 53 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "18ca5423-7532-4b79-b0e3-838d100510d1" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.60:443 10.132.0.20:46274 - v1-completions-path 2026-10-30T06:26:33.977380Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:26:36.383795Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:26:39.346367Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:26:49.658434Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:26:55.847668Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-10-30T06:26:55.615Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 57 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "605c787b-152a-4c15-8147-4bbdbc1e5d99" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.60:443 10.132.0.20:41590 - v1-completions-path [2026-10-30T06:26:55.707Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 58 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "44e18529-2a55-4638-b85f-24a9a843c650" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:55400 10.132.0.60:443 10.132.0.20:41592 - v1-completions-path [2026-10-30T06:26:55.788Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "e27a42a5-3c97-4992-938e-4a06b25f94bb" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:52002 10.132.0.60:443 10.132.0.20:43018 - v1-completions-path [2026-10-30T06:26:56.588Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 57 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "611e61b6-0c42-4a24-8a47-d13d5ffd69eb" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:52002 10.132.0.60:443 10.132.0.20:43026 - v1-completions-path [2026-10-30T06:26:56.740Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 55 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "4dfc3d75-89b6-4590-a9b0-373b53c401a5" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:55400 10.132.0.60:443 10.132.0.20:43038 - v1-completions-path 2026-10-30T06:27:12.679753Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:27:23.116433Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:27:27.257086Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:27:32.343Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "6c636229-f463-49cd-a79f-41808f86fa85" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:52002 10.132.0.60:443 10.132.0.20:35192 - v1-completions-path 2026-10-30T06:27:44.707510Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:27:46.779735Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:28:06.214423Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:28:10.711232Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:28:24.924Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 65 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "8400beeb-fcc1-4a8a-8075-847387ff2025" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:37220 10.132.0.60:443 10.132.0.20:37948 - v1-completions-path 2026-10-30T06:28:26.337803Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:28:28.415226Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:28:45.367Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 288 65 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "f632207c-b2cf-4dcc-98f7-7b6bffabd899" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:50460 10.132.0.60:443 10.132.0.20:40598 - v1-completions-path 2026-10-30T06:28:46.451128Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:28:57.379511Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:29:00.494425Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:29:18.307451Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:29:20.897945Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:29:41.927362Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:29:44.620649Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:31:20.673429Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:31:24.029555Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:32:54.461629Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:32:58.171687Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:32:58.171720Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:32:58.171731Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:32:58.171735Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T06:32:58.098Z] "DELETE /v1/api-keys/e2e-trlp-test-key HTTP/1.1" 404 - via_upstream - "-" 0 29 73 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "a221f7d7-757e-4b5f-a1a1-a45dbdb3fb9f" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.60:57670 10.132.0.60:443 10.132.0.20:42198 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:33:09.975824Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:33:12.057230Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:33:12.801Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 278 81 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "30152b45-1ab6-47bf-a32b-1746a25a27a8" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:50460 10.132.0.60:443 10.132.0.20:40500 - v1-completions-path 2026-10-30T06:33:15.105267Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T06:33:15.105294Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T06:33:15.105298Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T06:33:15.105303Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-30T06:33:15.070Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 35 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "4a38b332-c92e-4011-a447-54c5dc904dae" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.60:443 10.132.0.20:40514 - v1-completions-path 2026-10-30T06:33:27.082863Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:34:45.045726Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:35:00.398836Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:35:16.405646Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:35:16.738754Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:35:16.738873Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:35:16.738932Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:35:16.738964Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T06:35:16.608Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 80 322 130 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "90febe39-2562-4c6b-8136-b2d3b3cba0bc" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.60:59882 10.132.0.60:443 10.132.0.20:52304 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:35:30.633286Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:35:34.397558Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:35:34.397594Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:35:34.397608Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T06:35:34.397613Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T06:35:34.288Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 84 327 108 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "d079169a-4df4-409e-a3c7-ea379b059d4d" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.60:59882 10.132.0.60:443 10.132.0.20:49726 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:35:34.499862Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:35:46.401287Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:35:50.037638Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:35:57.665Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 62 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "7a86c587-1959-49d6-8c6b-63a96202e4b9" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:52002 10.132.0.60:443 10.132.0.20:49116 - v1-completions-path 2026-10-30T06:36:09.873625Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:36:17.301110Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T06:36:17.301136Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T06:36:17.301139Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T06:36:17.301143Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-30T06:36:17.237Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 63 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "f9753472-8284-4a88-80ad-a69c04ee9b30" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.60:443 10.132.0.20:52386 - v1-catch-all-path 2026-10-30T06:36:19.580512Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:36:25.089096Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:36:25.089136Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T06:36:25.089150Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T06:36:25.089154Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T06:36:25.007Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 81 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "cdd9d1e5-0f56-4ced-974b-f383a0f5e019" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.134.1.75:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.60:45670 10.132.0.60:443 10.132.0.20:52390 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:36:27.715142Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:36:33.747080Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-10-30T06:36:33.663Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 84 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "2cb63510-e7fd-45aa-a65f-b64a46c1cc35" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:33562 10.132.0.60:443 10.132.0.20:38940 - v1-catch-all-path 2026-10-30T06:36:41.673703Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:36:43.726193Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:36:46.647847Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-10-30T06:36:46.579Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 68 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "2707efe6-d5a9-4d9a-b1db-3b5735634e1a" "maas.apps.3b714319aaccfb1f8a93.prod.konflux-ocp-ci.dev" "10.132.0.53:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.60:33562 10.132.0.60:443 10.132.0.20:53174 - v1-catch-all-path 2026-10-30T06:36:58.476227Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:37:09.528753Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8