2026-11-04T01:16:42.038063Z info FLAG: --concurrency="0" 2026-11-04T01:16:42.038099Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-04T01:16:42.038106Z info FLAG: --help="false" 2026-11-04T01:16:42.038110Z info FLAG: --log_as_json="false" 2026-11-04T01:16:42.040170Z info FLAG: --log_caller="" 2026-11-04T01:16:42.040191Z info FLAG: --log_output_level="default:info" 2026-11-04T01:16:42.040195Z info FLAG: --log_stacktrace_level="default:none" 2026-11-04T01:16:42.040205Z info FLAG: --log_target="[stdout]" 2026-11-04T01:16:42.040209Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-04T01:16:42.040214Z info FLAG: --outlierLogPath="" 2026-11-04T01:16:42.040224Z info FLAG: --profiling="true" 2026-11-04T01:16:42.040228Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-04T01:16:42.040231Z info FLAG: --proxyLogLevel="warning" 2026-11-04T01:16:42.040234Z info FLAG: --serviceCluster="istio-proxy" 2026-11-04T01:16:42.040243Z info FLAG: --stsPort="0" 2026-11-04T01:16:42.040247Z info FLAG: --templateFile="" 2026-11-04T01:16:42.040250Z info FLAG: --tokenManagerPlugin="" 2026-11-04T01:16:42.040255Z info FLAG: --vklog="0" 2026-11-04T01:16:42.040259Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-04T01:16:42.041193Z info Proxy role ips=[10.132.0.54] type=router id=maas-default-gateway-openshift-default-75d7c785f-xkmtt.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-04T01:16:42.041572Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-04T01:16:42.043282Z info cpu limit detected as 2, setting concurrency 2026-11-04T01:16:42.043537Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-04T01:16:42.043550Z info JWT policy is third-party-jwt 2026-11-04T01:16:42.043554Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-04T01:16:42.045347Z info Starting default Istio SDS Server 2026-11-04T01:16:42.045374Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-04T01:16:42.045347Z info Opening status port 15020 2026-11-04T01:16:42.045397Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-04T01:16:42.046057Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-04T01:16:42.048584Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-04T01:16:42.051765Z info sds Starting SDS grpc server 2026-11-04T01:16:42.051802Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-04T01:16:42.052801Z info Starting proxy agent 2026-11-04T01:16:42.052844Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-04T01:16:42.130438Z info cache generated new workload certificate resourceName=default latency=83.237681ms ttl=23h59m59.869566733s 2026-11-04T01:16:42.130491Z info cache Root cert has changed, start rotating root cert 2026-11-04T01:16:42.130533Z info cache returned workload trust anchor from cache ttl=23h59m59.86946822s 2026-11-04T01:16:42.235331Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-04T01:16:42.380528Z info ads ADS: new connection for node:1 2026-11-04T01:16:42.380603Z info cache returned workload certificate from cache ttl=23h59m59.619399224s 2026-11-04T01:16:42.380988Z info ads ADS: new connection for node:2 2026-11-04T01:16:42.381580Z info cache returned workload trust anchor from cache ttl=23h59m59.618421642s 2026-11-04T01:16:42.460797Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:16:50.759901Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:16:51.980196Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-04T01:16:52.798690Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:16:53.761508Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-04T01:16:54.019680Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-04T01:16:54.987057Z info Readiness succeeded in 12.954507554s 2026-11-04T01:16:54.987392Z info Envoy proxy is ready 2026-11-04T01:17:03.862829Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:17:04.327762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-04T01:17:40.104Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 521 82 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "92ce841d-9a20-4001-8de8-5e48b64e57f0" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:35908 10.132.0.54:443 10.132.0.15:50080 - v1-model-routing 2026-11-04T01:17:41.656007Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:17:41.656048Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:17:41.656061Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T01:17:41.656066Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T01:17:41.592Z] "DELETE /v1/api-keys/59664447-adbb-4c01-9ca3-2c10be106cee HTTP/1.1" 200 - via_upstream - "-" 0 365 63 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "97596d0c-758e-46aa-a5b3-740358d5b9e6" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.134.0.21:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:44010 10.132.0.54:443 10.132.0.15:50082 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:19:58.613619Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:19:58.613725Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:19:58.613772Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:19:58.613800Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-04T01:19:58.519Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 94 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "18eb6f25-e1a3-4774-9a8c-7e8ec269bd1d" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.134.0.21:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:58744 10.132.0.54:443 10.132.0.15:41386 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:20:00.054269Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:20:00.054302Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:20:00.054315Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T01:20:00.054319Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T01:19:59.728Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 39 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "08618e38-96d0-401c-a650-40319f4fbea2" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.132.0.54:443 10.132.0.15:41404 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:19:59.954Z] "DELETE /v1/api-keys/4222b041-e944-4ea6-b94e-dd8fa5ac9a61 HTTP/1.1" 200 - via_upstream - "-" 0 321 100 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "e8d0aab2-bf61-48c7-956e-46d145e77c7a" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.134.0.21:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:58744 10.132.0.54:443 10.132.0.15:41410 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:20:01.747911Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T01:20:01.747951Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T01:20:01.747968Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T01:20:01.747975Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=27 [2026-11-04T01:20:01.672Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 75 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "324fa5c8-286a-4113-b981-8e7c432704ce" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.134.0.21:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:58752 10.132.0.54:443 10.132.0.15:41412 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:20:05.038944Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=27 2026-11-04T01:20:05.038969Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=27 2026-11-04T01:20:05.038971Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=27 2026-11-04T01:20:05.038975Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=27 [2026-11-04T01:20:05.002Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 77 36 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "6b318e05-11d3-4132-9137-7e9934830a66" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.54:443 10.132.0.15:41432 - v1-completions-path 2026-11-04T01:20:08.479031Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-04T01:20:08.414Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 283 73 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "a29e578d-e5e6-4101-be77-76e3a3b16fe4" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:35908 10.132.0.54:443 10.132.0.15:59748 - v1-completions-path 2026-11-04T01:20:16.775524Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-04T01:20:16.133Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 65 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "51203fcc-5201-48d3-8b9f-0dd6a382cc5a" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.54:443 10.132.0.15:59780 - v1-completions-path 2026-11-04T01:20:30.145942Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:25:16.090345Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:25:30.721355Z warn xdsproxy sendDownstreamDelta took 14.687509355s 2026-11-04T01:26:06.314956Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:26:17.586486Z warn xdsproxy sendDownstreamDelta took 10.15312956s 2026-11-04T01:27:25.116703Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:29:23.849883Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:31:27.263192Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:32:03.327105Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:33:26.907574Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:35:02.876728Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:36:04.720031Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:36:27.144728Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:36:38.502751Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:36:38.502781Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:36:38.502791Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:36:38.502796Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-04T01:36:38.442Z] "POST /v1/api-keys HTTP/1.1" 400 - via_upstream - "-" 80 94 60 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "5d047f9c-bbcf-4ac5-af93-4608b2bd0542" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.134.0.21:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:52826 10.132.0.54:443 10.132.0.15:44648 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:37:14.374960Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-04T01:37:18.166Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 77 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "a559a272-f328-4a69-8b75-f9f06340b9cf" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:60378 10.132.0.54:443 10.132.0.15:49156 - v1-completions-path 2026-11-04T01:37:30.438511Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:37:32.147106Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:37:37.774284Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:37:48.414312Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:37:52.157868Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:38:00.403940Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:38:04.255863Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T01:38:04.255911Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T01:38:04.255931Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T01:38:04.255940Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=27 [2026-11-04T01:38:04.171Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 84 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "9a7922b2-cf66-931b-bbc8-52df68696d8a" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.134.0.21:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:46650 10.132.0.54:443 10.132.0.15:56490 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:38:06.495958Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=27 [2026-11-04T01:38:06.081Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 65 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "a43fa3b0-d136-45a5-815f-2a3d31cb186c" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:57742 10.132.0.54:443 10.132.0.15:56506 - v1-completions-path [2026-11-04T01:38:06.441Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 60 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "e6bb379c-1fea-46b6-86b6-3b6521cd13e8" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:57742 10.132.0.54:443 10.132.0.15:56510 - v1-completions-path [2026-11-04T01:38:06.894Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 61 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "f1ebb72d-ec65-4428-a8a7-a66da52f290f" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:60378 10.132.0.54:443 10.132.0.15:56514 - v1-completions-path [2026-11-04T01:38:07.009Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 58 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "4e60e9e8-1de7-4eb3-afba-c78608b136d8" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:57742 10.132.0.54:443 10.132.0.15:56526 - v1-completions-path [2026-11-04T01:38:08.016Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 298 56 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "7f5cae36-b5e1-41f5-918e-f161261124d1" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:60378 10.132.0.54:443 10.132.0.15:56534 - v1-completions-path 2026-11-04T01:38:22.965977Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:38:33.090787Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:38:35.323182Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-04T01:38:41.207Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 62 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "040ed399-706e-43dd-970f-c81164477189" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:60378 10.132.0.54:443 10.132.0.15:56074 - v1-completions-path [2026-11-04T01:38:42.000Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "84a2165e-a975-40c9-91ea-d7f07522d267" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.133.0.49:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:60378 10.132.0.54:443 10.132.0.15:56096 - v1-completions-path 2026-11-04T01:38:54.011718Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:38:55.882937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:39:01.788494Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:39:13.260516Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:39:26.893554Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:39:28.523173Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:39:34.520064Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:39:52.084407Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:39:55.379449Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:40:10.510642Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:40:12.099863Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:40:17.951576Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:40:33.551248Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:40:43.803158Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:40:59.453089Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:41:02.159454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:43:17.012101Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:43:20.148165Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:44:21.288675Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T01:44:21.288724Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T01:44:21.288748Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=27 2026-11-04T01:44:21.288755Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=27 [2026-11-04T01:44:21.228Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 313 60 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "492a9f7a-65a9-46e8-8aab-9bf946f07c74" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.134.0.21:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:47542 10.132.0.54:443 10.132.0.15:44902 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:44:34.314547Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-04T01:44:40.041336Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:44:55.595141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:44:57.412137Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:45:03.759023Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:47:22.272373Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:47:36.002156Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:47:51.784044Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:48:03.540400Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:48:07.518759Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:48:15.424937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:48:18.043131Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:48:33.749793Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:51:05.695305Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=27 2026-11-04T01:51:05.695330Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=27 2026-11-04T01:51:05.695333Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=27 2026-11-04T01:51:05.695337Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=27 [2026-11-04T01:51:05.636Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 58 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "2999bfca-0da7-4eef-b116-e70934fec1f9" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.54:443 10.132.0.15:46882 - v1-catch-all-path 2026-11-04T01:51:08.038794Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:51:14.056034Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:51:17.925679Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=27 2026-11-04T01:51:18.153728Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=27 [2026-11-04T01:51:17.845Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 80 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "0c199bc2-8f1b-4355-89bc-43a717e77f42" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.133.0.52:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:37800 10.132.0.54:443 10.132.0.15:43980 - v1-catch-all-path [2026-11-04T01:51:18.088Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 65 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "c6d7d70b-f5fe-4ae7-9f56-8f9383139041" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.133.0.52:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:37800 10.132.0.54:443 10.132.0.15:43986 - v1-catch-all-path 2026-11-04T01:51:29.832841Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:51:31.525193Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:51:34.892612Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:51:34.892647Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:51:34.892661Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T01:51:34.892666Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T01:51:34.761Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 131 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "8d22c1ef-8d64-4af3-9dd2-2af22c9339db" "maas.apps.db7c6f5a60cadf03a449.prod.konflux-ocp-ci.dev" "10.134.0.140:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:56508 10.132.0.54:443 10.132.0.15:56232 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:51:46.439563Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T01:51:57.239480Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9