2026-10-30T04:54:12.429103Z info FLAG: --concurrency="0" 2026-10-30T04:54:12.429120Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T04:54:12.429124Z info FLAG: --help="false" 2026-10-30T04:54:12.429126Z info FLAG: --log_as_json="false" 2026-10-30T04:54:12.429128Z info FLAG: --log_caller="" 2026-10-30T04:54:12.429130Z info FLAG: --log_output_level="default:info" 2026-10-30T04:54:12.429133Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T04:54:12.429137Z info FLAG: --log_target="[stdout]" 2026-10-30T04:54:12.429139Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T04:54:12.429141Z info FLAG: --outlierLogPath="" 2026-10-30T04:54:12.429144Z info FLAG: --profiling="true" 2026-10-30T04:54:12.429146Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T04:54:12.429148Z info FLAG: --proxyLogLevel="warning" 2026-10-30T04:54:12.429149Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T04:54:12.429151Z info FLAG: --stsPort="0" 2026-10-30T04:54:12.429153Z info FLAG: --templateFile="" 2026-10-30T04:54:12.429155Z info FLAG: --tokenManagerPlugin="" 2026-10-30T04:54:12.429162Z info FLAG: --vklog="0" 2026-10-30T04:54:12.429165Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T04:54:12.429306Z info Proxy role ips=[10.133.0.37] type=router id=maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T04:54:12.429353Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T04:54:12.430400Z info cpu limit detected as 2, setting concurrency 2026-10-30T04:54:12.430593Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T04:54:12.430604Z info JWT policy is third-party-jwt 2026-10-30T04:54:12.430607Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T04:54:12.432457Z info Starting default Istio SDS Server 2026-10-30T04:54:12.432476Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T04:54:12.432499Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T04:54:12.432820Z info Opening status port 15020 2026-10-30T04:54:12.434292Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T04:54:12.435738Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T04:54:12.436660Z info Starting proxy agent 2026-10-30T04:54:12.436723Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T04:54:12.445744Z info sds Starting SDS grpc server 2026-10-30T04:54:12.445809Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T04:54:12.513974Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T04:54:12.615019Z info ads ADS: new connection for node:1 2026-10-30T04:54:12.615136Z info ads ADS: new connection for node:2 2026-10-30T04:54:12.632308Z info cache generated new workload certificate resourceName=default latency=198.293521ms ttl=23h59m59.367695937s 2026-10-30T04:54:12.632342Z info cache Root cert has changed, start rotating root cert 2026-10-30T04:54:12.632372Z info cache returned workload certificate from cache ttl=23h59m59.367632548s 2026-10-30T04:54:12.632383Z info cache returned workload trust anchor from cache ttl=23h59m59.367617812s 2026-10-30T04:54:12.632466Z info cache returned workload trust anchor from cache ttl=23h59m59.36753523s 2026-10-30T04:54:12.632783Z info cache returned workload trust anchor from cache ttl=23h59m59.367217815s 2026-10-30T04:54:12.676436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:54:18.623386Z info Readiness succeeded in 6.197033559s 2026-10-30T04:54:18.623695Z info Envoy proxy is ready 2026-10-30T04:54:27.971768Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:03.852729Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:06.006308Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:08.121000Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:10.495625Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:13.628152Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:17.111192Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:55:33.671025Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T04:56:08.716Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 347 84 - "52.71.36.72,10.132.0.15" "curl/7.76.1" "96cfeba5-8cd9-408e-87ea-e96787918b47" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:38984 10.133.0.37:443 10.132.0.15:46452 - v1-model-routing 2026-10-30T04:56:10.034400Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:56:10.034437Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:56:10.034451Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:56:10.034455Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T04:56:09.959Z] "DELETE /v1/api-keys/5464bafa-b281-492e-b76b-6db806303b00 HTTP/1.1" 200 - via_upstream - "-" 0 365 75 - "52.71.36.72,10.132.0.15" "curl/7.76.1" "0f369b5e-4f12-4fe7-a4de-6688bef10380" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:55016 10.133.0.37:443 10.132.0.15:46460 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:56:10.195040Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:56:10.195077Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:56:10.195093Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T04:56:10.195097Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T04:56:10.124Z] "GET /v1/subscriptions HTTP/1.1" 200 - via_upstream - "-" 0 488 71 - "52.71.36.72,10.132.0.15" "curl/7.76.1" "e8c3ff66-8454-48ae-a766-89670e6b8169" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:55026 10.133.0.37:443 10.132.0.15:46464 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:58:34.911264Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:34.911311Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:34.911372Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:34.911384Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T04:58:34.796Z] "DELETE /v1/api-keys/653d1a5f-a107-4c4f-b565-a1defd7f54ac HTTP/1.1" 200 - via_upstream - "-" 0 321 114 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "6ba9c142-91a2-4a8b-9e4a-77a50dbcc360" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:34906 10.133.0.37:443 10.132.0.15:45694 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:58:35.864412Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:58:35.864458Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:58:35.864479Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T04:58:35.864485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:58:35.959686Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:58:35.959718Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:58:35.959732Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T04:58:35.959737Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T04:58:35.788Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 75 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "3fec95aa-3f60-4fbb-bb31-a1b2a109f5c7" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:34916 10.133.0.37:443 10.132.0.15:45696 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T04:58:35.883Z] "DELETE /v1/api-keys/8126ceb0-ed76-43d6-b4d2-a183b096b5bf HTTP/1.1" 200 - via_upstream - "-" 0 321 75 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "c1f679cc-e02c-4f66-b955-e76e0c353cfb" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:34916 10.133.0.37:443 10.132.0.15:45698 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:58:36.515042Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:36.515077Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:36.515091Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T04:58:36.515096Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T04:58:36.433Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 81 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "87645010-e081-4111-9f7e-59b7e9d96b86" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:34906 10.133.0.37:443 10.132.0.15:45708 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:58:49.816791Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:49.816827Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:49.816841Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:49.816846Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T04:58:49.729Z] "DELETE /v1/api-keys/9a0a3310-c31a-4dbc-8c40-5549c432e668 HTTP/1.1" 200 - via_upstream - "-" 0 321 87 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "6588a348-4164-40f5-9b5e-67c451b6405a" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:34906 10.133.0.37:443 10.132.0.15:43742 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T04:58:49.837Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 49 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "643c172a-b718-4b6a-a04c-bb48a8414dab" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.37:443 10.132.0.15:43750 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:58:50.092444Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:50.092478Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:50.092493Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T04:58:50.092498Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T04:58:49.996Z] "DELETE /v1/api-keys/b0982aef-99dc-43a6-90d0-b666df037295 HTTP/1.1" 200 - via_upstream - "-" 0 321 96 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "56535aeb-be11-4710-86dc-99c802fbb09d" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:34906 10.133.0.37:443 10.132.0.15:43752 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T04:58:54.132758Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-10-30T04:58:54.067Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 281 66 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "0c3c0742-f042-4957-b7ea-e54b2610a712" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:38984 10.133.0.37:443 10.132.0.15:43770 - v1-completions-path [2026-10-30T04:58:54.164Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 66 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "5487c81a-0af5-411b-b510-b2fe91f0dc52" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:38984 10.133.0.37:443 10.132.0.15:43782 - v1-completions-path 2026-10-30T04:58:58.562994Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T04:59:14.347519Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:02:50.536517Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: gRPC status code is not OK thread=24 [2026-10-30T05:02:48.516Z] "GET /maas-api/v1/models HTTP/1.1" 500 - - "-" 0 23 2020 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "4abfe442-39ed-4a9b-9bbf-85cfebd5525f" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.37:443 10.132.0.15:44340 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T05:03:00.185Z] "GET /v1/tenants HTTP/1.1" 404 NR route_not_found - "-" 0 0 12 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "30c3161a-141d-4dc2-abf0-e6c8b3bfac4e" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "-" - - 10.133.0.37:443 10.132.0.15:52832 - - 2026-10-30T05:03:39.404274Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:04:07.415839Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:05:38.081861Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:08:37.982260Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:09:16.874684Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:10:54.868230Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:22:27.062250Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T05:37:33.332393Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-10-30T05:37:33.357543Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:37:33.357576Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:37:33.357590Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:37:33.357595Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T05:37:33.260Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 69 312 97 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "d21a1ca6-1ed0-4454-8361-31c9de9fc757" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:40918 10.133.0.37:443 10.132.0.15:41048 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:40:49.087273Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:40:51.420596Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:40:51.420633Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:40:51.420648Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:40:51.420653Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T05:40:51.343Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 77 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "a461941d-b38b-4a5b-b603-39165df75028" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:59590 10.133.0.37:443 10.132.0.15:49562 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:41:15.926194Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:41:54.964923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:42:00.855Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 82 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "11accdf7-5191-4e54-b575-769c8894f23f" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:53916 10.133.0.37:443 10.132.0.15:60766 - v1-completions-path 2026-10-30T05:42:14.011672Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:42:18.152695Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:42:21.915414Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:42:21.915451Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:42:21.915466Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:42:21.915471Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T05:42:21.816Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 93 336 99 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "ae9a463d-2d08-4668-92b2-7181e7d7eb46" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:45222 10.133.0.37:443 10.132.0.15:60580 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T05:42:21.962Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 85 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "6c4bc040-2d6e-4867-af68-01c89ea29d01" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:46690 10.133.0.37:443 10.132.0.15:60588 - v1-completions-path 2026-10-30T05:42:34.422088Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:42:50.155665Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:42:55.615696Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:42:55.615737Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:42:55.615757Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T05:42:55.615763Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T05:42:55.538Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 76 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "070335f8-45e7-4824-9943-20ef3d8c9d0c" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:34526 10.133.0.37:443 10.132.0.15:60924 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T05:42:58.144Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 59 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "19a9b5e7-cf9c-405c-b981-8c57168c04ee" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:53916 10.133.0.37:443 10.132.0.15:60932 - v1-completions-path [2026-10-30T05:42:58.407Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 57 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "6a618694-9df5-42aa-a5f0-910eac85ae5a" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:53916 10.133.0.37:443 10.132.0.15:60940 - v1-completions-path [2026-10-30T05:42:58.491Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 52 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "5ff7250f-754e-4874-9eb7-e692e9d25077" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:46690 10.133.0.37:443 10.132.0.15:60948 - v1-completions-path 2026-10-30T05:43:14.304011Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:43:24.684491Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:43:29.127460Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T05:43:34.084Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 61 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "4ae23994-d431-4473-9d92-e4fd61dae51a" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:53916 10.133.0.37:443 10.132.0.15:60404 - v1-completions-path [2026-10-30T05:43:34.754Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 55 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "601a8f56-e16c-4292-8999-bf14138fa5e9" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:53916 10.133.0.37:443 10.132.0.15:45668 - v1-completions-path 2026-10-30T05:43:46.827508Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:43:48.987277Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:44:05.516805Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:44:10.274588Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:44:27.598992Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:44:44.720117Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:44:55.378257Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:44:58.182674Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:45:16.411374Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:45:19.023437Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:45:24.098134Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:45:24.098171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:45:24.098185Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:45:24.098190Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T05:45:24.025Z] "DELETE /v1/api-keys/1d139a35-72d9-4303-ba57-81deb16dfa92 HTTP/1.1" 200 - via_upstream - "-" 0 375 72 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "79978c96-9eb9-46c0-be99-89f7c6e98b55" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:54878 10.133.0.37:443 10.132.0.15:35766 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T05:47:11.526Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 45 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "1ccb64e2-5e16-4cc8-8918-00594dfc5f9b" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.37:443 10.132.0.15:39258 - v1-completions-path 2026-10-30T05:48:42.690407Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:48:46.155015Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:48:46.155050Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:48:46.155067Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:48:46.155073Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T05:48:46.078Z] "DELETE /v1/api-keys/e2e-trlp-test-key HTTP/1.1" 404 - via_upstream - "-" 0 29 76 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "8986c169-c2b9-49a4-853e-2a84b8c8f58f" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.132.0.45:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.37:46456 10.133.0.37:443 10.132.0.15:44108 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:48:57.925468Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:48:59.887683Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:49:14.471434Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:50:31.470180Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:50:49.438330Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:51:04.367866Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:51:18.466719Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:51:22.232780Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:51:33.721631Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:51:37.197384Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:51:57.117219Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:52:00.298355Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-10-30T05:52:06.961138Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:52:12.601688Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-10-30T05:52:12.527Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 73 - "52.71.36.72,10.132.0.15" "python-requests/2.34.2" "af99c276-9286-4d9c-87ef-c04741486c4c" "maas.apps.70853b44304d2f3542e7.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.37:42844 10.133.0.37:443 10.132.0.15:53610 - v1-catch-all-path 2026-10-30T05:52:15.319637Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:52:33.118568Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:52:35.360618Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:52:49.884704Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T05:53:00.874059Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8