2026-10-30T21:57:28.003628Z info FLAG: --concurrency="0" 2026-10-30T21:57:28.003649Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T21:57:28.003653Z info FLAG: --help="false" 2026-10-30T21:57:28.003656Z info FLAG: --log_as_json="false" 2026-10-30T21:57:28.003658Z info FLAG: --log_caller="" 2026-10-30T21:57:28.003660Z info FLAG: --log_output_level="default:info" 2026-10-30T21:57:28.003662Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T21:57:28.003666Z info FLAG: --log_target="[stdout]" 2026-10-30T21:57:28.003669Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T21:57:28.003671Z info FLAG: --outlierLogPath="" 2026-10-30T21:57:28.003673Z info FLAG: --profiling="true" 2026-10-30T21:57:28.003675Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T21:57:28.003677Z info FLAG: --proxyLogLevel="warning" 2026-10-30T21:57:28.003679Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T21:57:28.003681Z info FLAG: --stsPort="0" 2026-10-30T21:57:28.003683Z info FLAG: --templateFile="" 2026-10-30T21:57:28.003685Z info FLAG: --tokenManagerPlugin="" 2026-10-30T21:57:28.003688Z info FLAG: --vklog="0" 2026-10-30T21:57:28.003690Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T21:57:28.003862Z info Proxy role ips=[10.134.0.36] type=router id=maas-default-gateway-openshift-default-6cc458d887-f89xr.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T21:57:28.003914Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T21:57:28.005317Z info cpu limit detected as 2, setting concurrency 2026-10-30T21:57:28.005584Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T21:57:28.005597Z info JWT policy is third-party-jwt 2026-10-30T21:57:28.005600Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T21:57:28.007872Z info Starting default Istio SDS Server 2026-10-30T21:57:28.007911Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T21:57:28.007943Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T21:57:28.008726Z info Opening status port 15020 2026-10-30T21:57:28.009488Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T21:57:28.010156Z info sds Starting SDS grpc server 2026-10-30T21:57:28.010176Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T21:57:28.011645Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T21:57:28.012999Z info Starting proxy agent 2026-10-30T21:57:28.013041Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T21:57:28.111175Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T21:57:28.114972Z info cache generated new workload certificate resourceName=default latency=104.644476ms ttl=23h59m59.885032164s 2026-10-30T21:57:28.115059Z info cache Root cert has changed, start rotating root cert 2026-10-30T21:57:28.115111Z info cache returned workload trust anchor from cache ttl=23h59m59.88488977s 2026-10-30T21:57:28.277723Z info ads ADS: new connection for node:1 2026-10-30T21:57:28.277791Z info cache returned workload certificate from cache ttl=23h59m59.722210346s 2026-10-30T21:57:28.278157Z info ads ADS: new connection for node:2 2026-10-30T21:57:28.278230Z info cache returned workload trust anchor from cache ttl=23h59m59.721771241s 2026-10-30T21:57:28.369639Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T21:57:36.292887Z info Readiness succeeded in 8.292601418s 2026-10-30T21:57:36.293281Z info Envoy proxy is ready [2026-10-30T21:58:04.476Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 365 83 - "52.71.36.72,10.132.0.11" "curl/7.76.1" "fd96d149-8a6f-434e-a9da-bc2f5f73e4f3" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.38:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:33556 10.134.0.36:443 10.132.0.11:55390 - v1-model-routing 2026-10-30T22:03:11.182410Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:03:11.182447Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:03:11.182461Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:03:11.182465Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T22:03:11.062Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 120 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "172399a4-0dc4-489c-b5a6-f8bd16c7d787" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.36:38726 10.134.0.36:443 10.132.0.11:37390 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T22:03:13.178233Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:03:13.178267Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:03:13.178281Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T22:03:13.178286Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T22:03:12.651Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 87 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "6fcfc514-258d-489f-9cab-ceb9a377ee25" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.36:443 10.132.0.11:37406 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T22:03:13.065Z] "DELETE /v1/api-keys/ebea3dc9-8273-4d3d-8bee-0f4e3c41f340 HTTP/1.1" 200 - via_upstream - "-" 0 321 113 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "5881c067-32f8-4c47-86aa-f9332d634c53" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.36:38726 10.134.0.36:443 10.132.0.11:37420 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T22:03:13.675014Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:03:13.675048Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:03:13.675061Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T22:03:13.675066Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T22:03:13.566Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 108 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "3c87c78d-d0b8-4d1d-95d1-65948795e6d6" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.36:38726 10.134.0.36:443 10.132.0.11:37422 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T22:03:25.681Z] "POST /maas-api/internal/v1/api-keys/cleanup HTTP/1.1" 404 NR route_not_found - "-" 2 0 10 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "5193da69-6fff-422c-bca0-5c189033d386" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "-" - - 10.134.0.36:443 10.132.0.11:47074 - - 2026-10-30T22:09:53.880120Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:09:53.880171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:09:53.880190Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T22:09:53.880194Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:09:54.205704Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:09:54.205741Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:09:54.205755Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:09:54.205760Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T22:09:53.538Z] "POST /maas-api/v1/tokens HTTP/1.1" 401 - - "-" 20 88 23 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "884b1a9c-b4b0-483e-94c3-5ef49356ecc5" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.36:443 10.132.0.11:60010 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T22:09:53.767Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 112 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "cf4e8416-21e5-42e3-9a6d-ab106ddaca8a" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.36:58918 10.134.0.36:443 10.132.0.11:60012 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T22:09:54.087Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 79 321 117 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "2112b784-4bf1-4350-b2f1-3063184f4749" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.36:58922 10.134.0.36:443 10.132.0.11:60026 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T22:09:54.740274Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-30T22:09:55.003723Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:09:55.003759Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:09:55.003772Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T22:09:55.003776Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:09:55.003941Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-10-30T22:09:55.003995Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T22:09:55.004003Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T22:09:55.004005Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T22:09:55.004009Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T22:09:54.407Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 136 331 73 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "fa7ab3af-acb0-4ae2-8099-34ad94e5b6d2" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.38:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:39552 10.134.0.36:443 10.132.0.11:60042 - v1-chat-completions-path [2026-10-30T22:09:54.687Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 135 2568 58 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "22f719e7-720f-442d-8c57-0691579530ef" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.38:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:39552 10.134.0.36:443 10.132.0.11:60054 - v1-chat-completions-path [2026-10-30T22:09:54.948Z] "POST /v1/completions HTTP/1.1" 404 - via_upstream - "-" 94 126 55 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "7241a248-d5eb-449b-98be-c94ef633fa13" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.38:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:39552 10.134.0.36:443 10.132.0.11:60060 - v1-completions-path 2026-10-30T22:10:04.621342Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:10:04.621546Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T22:10:04.621606Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T22:10:04.621639Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T22:10:04.590Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 31 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "d683eba1-b511-49da-bb6c-1efb704a6392" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.36:58918 10.134.0.36:443 10.132.0.11:59874 - odh-ai-gateway-infra.maas-api-route.4 [2026-10-30T22:10:07.403Z] "GET /v1/tenants HTTP/1.1" 404 NR route_not_found - "-" 0 0 56 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "549c18d3-0e56-4a3f-ad2e-e9ea0d5a1a89" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "-" - - 10.134.0.36:443 10.132.0.11:34098 - - 2026-10-30T22:11:24.511523Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:12:07.052872Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:13:49.978425Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:14:30.079608Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:16:24.355490Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:17:00.570102Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:18:42.949491Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:20:33.335841Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:28:46.086297Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T22:37:45.372672Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:37:45.372709Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:37:45.372722Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T22:37:45.372726Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T22:37:45.193Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 30 314 179 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "fdcb4c41-7ba8-4aa3-bcfe-08240163bd5d" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.36:50024 10.134.0.36:443 10.132.0.11:60544 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T22:37:50.324440Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:37:50.324472Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:37:50.324485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:37:50.324489Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T22:37:50.958153Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:37:50.215Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 108 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "416d24d2-59de-46da-b3d9-11e601c55f9d" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.36:50024 10.134.0.36:443 10.132.0.11:49886 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T22:38:15.516635Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:38:34.856354Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:38:34.856409Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:38:34.856420Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T22:38:34.856425Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T22:38:34.687Z] "POST /v1/api-keys HTTP/1.1" 400 - via_upstream - "-" 80 94 168 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "b34cf919-b697-40da-83bc-4115838ff0d7" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.132.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.36:50024 10.134.0.36:443 10.132.0.11:33308 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T22:39:17.638810Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T22:39:17.638837Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T22:39:17.638840Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T22:39:17.638844Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T22:39:17.602Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 36 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "e8ef4041-de3c-482e-960c-eee1e4f37b11" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.36:443 10.132.0.11:54040 - v1-completions-path 2026-10-30T22:39:24.668911Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:39:32.444Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 58 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "af1abe80-5c12-44ff-8870-408c25a1793c" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.36:443 10.132.0.11:55696 - v1-completions-path 2026-10-30T22:39:46.036354Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:39:51.007239Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:40:02.849Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 299 110 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "8264ccb1-a60b-49d6-9e13-b32124d43ffe" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:50604 10.134.0.36:443 10.132.0.11:43282 - v1-completions-path 2026-10-30T22:40:17.301279Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:40:34.261558Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:40:41.733238Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-30T22:40:41.673Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 63 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "1b7711e5-7374-4acd-a26d-768a776ae563" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:50604 10.134.0.36:443 10.132.0.11:42192 - v1-completions-path [2026-10-30T22:40:42.793Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 53 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "18269d13-74bd-4832-a0a5-43245475b8fe" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.36:443 10.132.0.11:42198 - v1-completions-path [2026-10-30T22:40:44.629Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 58 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "63b13347-11a9-4bc5-a1a3-bd4563d10dff" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:35344 10.134.0.36:443 10.132.0.11:42216 - v1-completions-path 2026-10-30T22:40:46.456679Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-10-30T22:40:45.895Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 57 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "eb98678e-fa17-4195-ac5c-01d44139a0b3" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:50604 10.134.0.36:443 10.132.0.11:42222 - v1-completions-path [2026-10-30T22:40:46.404Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 55 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "41e6f054-d1ee-46a9-9dab-cb1382220660" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:35344 10.134.0.36:443 10.132.0.11:39514 - v1-completions-path [2026-10-30T22:40:47.801Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 57 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "1f2c1260-2021-4131-814a-442df4ab2f02" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:35344 10.134.0.36:443 10.132.0.11:39520 - v1-completions-path 2026-10-30T22:41:04.920636Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:41:15.660344Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:41:22.076647Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:41:36.813Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 62 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "25923eb3-75e0-4676-b698-73858f40de5a" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:50604 10.134.0.36:443 10.132.0.11:35348 - v1-completions-path 2026-10-30T22:41:50.556624Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:42:02.308835Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:42:08.284Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 74 286 74 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "45dbe8d8-3dea-4bba-a916-b68e9cd578bb" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "10.133.0.52:8000" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local 10.134.0.36:42508 10.134.0.36:443 10.132.0.11:50540 - v1-completions-path 2026-10-30T22:42:21.912240Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:42:37.608943Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:42:59.857894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:43:41.923239Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:43:54.927349Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:44:01.164454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:44:32.638096Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:44:38.183746Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:44:47.917Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 33 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "5513a6ef-15f0-4ac2-a556-2fc663e0dd2c" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.36:443 10.132.0.11:41946 - v1-completions-path 2026-10-30T22:44:56.605289Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:45:38.270463Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:45:49.104831Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:46:17.995141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:46:24.007068Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:49:00.049572Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:49:06.212124Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:50:46.546838Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:51:04.009762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:51:08.616437Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:51:33.314519Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:53:42.277295Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:54:26.196561Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:54:56.605983Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:55:11.798955Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:55:20.601350Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:55:29.363088Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:55:35.643633Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T22:55:48.252Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 53 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "9d2f139e-3af1-4c6b-a951-a034187b7ba4" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.36:443 10.132.0.11:43754 - v1-completions-path 2026-10-30T22:56:02.088554Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:56:16.735428Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-10-30T22:59:39.250273Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T22:59:39.250302Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T22:59:39.250305Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T22:59:39.250309Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T22:59:39.089Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 160 - "52.71.36.72,10.132.0.11" "python-requests/2.34.2" "ca06e2c3-c4da-4dcc-931b-d09e124fa870" "maas.apps.3dc46e1647a5f0a10275.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.36:443 10.132.0.11:54058 - v1-catch-all-path 2026-10-30T22:59:45.158555Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T22:59:55.206868Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T23:00:18.782709Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T23:00:23.587056Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T23:00:37.961664Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T23:00:49.748632Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8