2026-11-03T01:25:26.366835Z info FLAG: --concurrency="0" 2026-11-03T01:25:26.366861Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T01:25:26.366867Z info FLAG: --help="false" 2026-11-03T01:25:26.366870Z info FLAG: --log_as_json="false" 2026-11-03T01:25:26.366873Z info FLAG: --log_caller="" 2026-11-03T01:25:26.366876Z info FLAG: --log_output_level="default:info" 2026-11-03T01:25:26.366879Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T01:25:26.366886Z info FLAG: --log_target="[stdout]" 2026-11-03T01:25:26.366889Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T01:25:26.366892Z info FLAG: --outlierLogPath="" 2026-11-03T01:25:26.366894Z info FLAG: --profiling="true" 2026-11-03T01:25:26.366897Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T01:25:26.366901Z info FLAG: --proxyLogLevel="warning" 2026-11-03T01:25:26.366904Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T01:25:26.366907Z info FLAG: --stsPort="0" 2026-11-03T01:25:26.366910Z info FLAG: --templateFile="" 2026-11-03T01:25:26.366913Z info FLAG: --tokenManagerPlugin="" 2026-11-03T01:25:26.366915Z info FLAG: --vklog="0" 2026-11-03T01:25:26.366919Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T01:25:26.367106Z info Proxy role ips=[10.133.0.169] type=router id=maas-default-gateway-openshift-default-7bdc57c4cf-rsqf2.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T01:25:26.367167Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T01:25:26.368694Z info cpu limit detected as 2, setting concurrency 2026-11-03T01:25:26.368984Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T01:25:26.368998Z info JWT policy is third-party-jwt 2026-11-03T01:25:26.369001Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T01:25:26.371391Z info Starting default Istio SDS Server 2026-11-03T01:25:26.371421Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T01:25:26.371445Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T01:25:26.371977Z info Opening status port 15020 2026-11-03T01:25:26.373251Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T01:25:26.373882Z info sds Starting SDS grpc server 2026-11-03T01:25:26.373921Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T01:25:26.375253Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T01:25:26.376404Z info Starting proxy agent 2026-11-03T01:25:26.376484Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T01:25:26.470991Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T01:25:26.613531Z info cache generated new workload certificate resourceName=default latency=239.531302ms ttl=23h59m59.386474496s 2026-11-03T01:25:26.613574Z info cache Root cert has changed, start rotating root cert 2026-11-03T01:25:26.613615Z info cache returned workload trust anchor from cache ttl=23h59m59.386386241s 2026-11-03T01:25:26.635225Z info ads ADS: new connection for node:1 2026-11-03T01:25:26.635442Z info cache returned workload certificate from cache ttl=23h59m59.364560371s 2026-11-03T01:25:26.635816Z info ads ADS: new connection for node:2 2026-11-03T01:25:26.636021Z info cache returned workload trust anchor from cache ttl=23h59m59.363981325s 2026-11-03T01:25:26.716265Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:25:35.300911Z info Readiness succeeded in 8.937335685s 2026-11-03T01:25:35.301270Z info Envoy proxy is ready 2026-11-03T01:25:36.321544Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:25:50.124759Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:26:00.331106Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:26:03.019637Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:26:03.019679Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:26:03.019695Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:26:03.019701Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T01:26:02.873Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 145 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "15b29e58-e9b1-49e5-b2a5-7e9a3b3168d5" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.169:54876 10.133.0.169:443 10.132.0.14:35958 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T01:26:03.326152Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T01:26:03.248Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 77 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "5bafb87c-427f-4b2b-bcce-9c98eb8f3bb5" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.132.0.52:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:53182 10.133.0.169:443 10.132.0.14:35976 - v1-catch-all-path 2026-11-03T01:26:17.066137Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:26:28.718923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:26:32.512343Z info ads ADS: new connection for node:3 2026-11-03T01:26:32.512543Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T01:26:32.512579Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T01:26:35.590424Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:27:14.663519Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:27:38.704496Z info ads ADS: "" 3 terminated with stream closed 2026-11-03T01:27:38.789699Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:27:45.208424Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:27:58.362489Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:30:32.253040Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:31:00.285257Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:31:26.121426Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:31:38.178551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:31:44.608157Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:31:51.901022Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:32:03.089337Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:32:09.270420Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:32:27.225305Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T01:32:43.303Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 278 76 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "6d03a29a-9a64-4f10-b84d-907d2b91df7b" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41790 10.133.0.169:443 10.132.0.14:53932 - v1-completions-path 2026-11-03T01:32:44.245488Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-03T01:32:44.186Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 59 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "0149bcb2-29ea-4a87-950b-004e8a0c8fe1" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41790 10.133.0.169:443 10.132.0.14:53948 - v1-completions-path 2026-11-03T01:33:14.768346Z info ads ADS: new connection for node:4 2026-11-03T01:33:14.773796Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T01:33:14.773843Z info cache read certificate from file resource=file-root:system 2026-11-03T01:33:18.702961Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:33:19.888062Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:34:28.040009Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:34:28.040042Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:34:28.040055Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:34:28.040061Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T01:34:28.040248Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T01:34:27.915Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1720 124 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "50964711-5540-4839-8330-51fe3bc1322f" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.169:46888 10.133.0.169:443 10.132.0.14:50184 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T01:34:41.951392Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:34:41.951427Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:34:41.951438Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:34:41.951443Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T01:34:41.951600Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T01:34:41.831Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1722 119 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "28b7b1a6-129f-4093-b414-a1278377fd7f" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.169:48232 10.133.0.169:443 10.132.0.14:33308 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T01:34:48.889959Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:34:48.889989Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:34:48.889997Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T01:34:48.890002Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T01:34:48.890144Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T01:34:48.781Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1722 108 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "ceee84a3-1ab7-4653-a20c-54da6749027b" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.169:46888 10.133.0.169:443 10.132.0.14:36266 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T01:35:08.738525Z info ads ADS: "" 4 terminated 2026-11-03T01:35:08.779187Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:35:11.453571Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:35:22.084028Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:35:40.427500Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:35:45.097963Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T01:35:49.736Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 305 82 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "ddfca5c1-596a-4aa7-9440-15d26c9b6ecf" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41686 10.133.0.169:443 10.132.0.14:47224 - v1-completions-path 2026-11-03T01:35:51.639952Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:35:51.639991Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:35:51.640010Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T01:35:51.640016Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T01:35:51.574Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 170 65 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "b8bc12c3-d440-417a-bd79-b4f80834f5b2" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41694 10.133.0.169:443 10.132.0.14:47240 - v1-catch-all-path 2026-11-03T01:36:05.173714Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:36:22.002880Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T01:36:27.768Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 56 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "aad3ed9c-1042-4337-ae9b-ab7c13900425" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.169:443 10.132.0.14:58798 - v1-completions-path [2026-11-03T01:36:28.858Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 59 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "665a7339-30de-4c19-81c4-f62373d8c5cb" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41694 10.133.0.169:443 10.132.0.14:58806 - v1-completions-path [2026-11-03T01:36:32.800Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 298 62 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "4f24a169-8f40-41fb-a99f-2f1383a8bfea" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41686 10.133.0.169:443 10.132.0.14:58830 - v1-completions-path 2026-11-03T01:36:49.808603Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:37:00.404435Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:37:06.796697Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:37:18.662923Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-03T01:37:18.657Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 71 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "e6659b5e-82e4-43ad-a7c2-fa5f28d0511b" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41694 10.133.0.169:443 10.132.0.14:35814 - v1-completions-path [2026-11-03T01:37:19.449Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 57 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "4f4f3451-1044-494d-91f2-e2f960f46555" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41686 10.133.0.169:443 10.132.0.14:35820 - v1-completions-path 2026-11-03T01:37:35.201348Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:37:39.787001Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:37:46.853068Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:37:46.853114Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:37:46.853135Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:37:46.853141Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T01:37:46.653Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 199 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "4f44c518-2138-4c96-b63e-5e6c6db30767" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.169:43864 10.133.0.169:443 10.132.0.14:59938 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T01:37:53.219128Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:38:06.242636Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:38:14.101258Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:38:24.229560Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:38:34.783258Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:38:52.320658Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:38:58.270369Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:39:16.365478Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:39:18.422983Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:39:18.423021Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:39:18.423036Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T01:39:18.423041Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T01:39:18.307Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 89 331 116 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "a9959e74-c03e-4885-b1c2-4c72e3fd9d44" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.169:58690 10.133.0.169:443 10.132.0.14:54150 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T01:39:21.906248Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T01:39:25.494Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 0 35 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "f88f4030-048e-4003-87b5-9c564276b6bb" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.169:443 10.132.0.14:33216 - v1-completions-path 2026-11-03T01:39:33.164058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:39:56.520518Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:40:07.232672Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T01:40:21.699Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 40 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "df2b3837-92b1-4dce-a277-13ebf62cdf87" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.169:443 10.132.0.14:36360 - v1-completions-path 2026-11-03T01:40:25.978104Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:40:33.678118Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:42:59.784649Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:44:29.103831Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:44:48.397526Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:44:53.185217Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:45:06.511026Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:45:22.651652Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:45:27.475025Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T01:45:30.524Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 82 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "9f95fb2b-4245-40d4-ab99-397babf8cf32" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41686 10.133.0.169:443 10.132.0.14:51880 - v1-completions-path 2026-11-03T01:45:47.183413Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:45:57.982922Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:46:04.271595Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T01:46:17.780Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 65 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "6286155a-ae2f-40b2-908e-a0678ce58dc5" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41694 10.133.0.169:443 10.132.0.14:39340 - v1-completions-path 2026-11-03T01:46:19.922707Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:46:19.922743Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:46:19.922759Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T01:46:19.922764Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T01:46:19.811Z] "DELETE /v1/api-keys/2a300ae6-e13a-44c1-84fd-55ad6a46929f HTTP/1.1" 200 - via_upstream - "-" 0 365 111 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "75844e6e-49ec-4ee6-b122-f072dbc8fcce" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.134.0.154:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.169:57994 10.133.0.169:443 10.132.0.14:39348 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T01:46:35.263581Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:46:39.696647Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:46:47.332473Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:46:54.202628Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:46:54.202664Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T01:46:54.202679Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T01:46:54.202684Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T01:46:54.091Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 66 308 111 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "bac3f283-98e2-4450-b857-eb89e4627858" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.179:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.169:38118 10.133.0.169:443 10.132.0.14:44972 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T01:46:57.361Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 70 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "4da5c949-8e50-41d4-916d-953732287869" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41694 10.133.0.169:443 10.132.0.14:55170 - v1-completions-path [2026-11-03T01:46:59.193Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 63 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "e4b9fea1-611c-4ec7-9d8e-868a2eb13683" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41694 10.133.0.169:443 10.132.0.14:55190 - v1-completions-path [2026-11-03T01:47:00.302Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 63 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "915d7be1-7062-4497-9f82-2536c76dbb68" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41694 10.133.0.169:443 10.132.0.14:55200 - v1-completions-path [2026-11-03T01:47:02.174Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 65 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "b538db3e-f020-4d9a-8ba0-6d10f21a5230" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:41694 10.133.0.169:443 10.132.0.14:55204 - v1-completions-path 2026-11-03T01:47:18.373867Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:47:28.976235Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:47:35.590760Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:47:41.467087Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:47:53.615072Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:48:17.068594Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:48:27.736027Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:48:32.507846Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:48:50.889107Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:48:56.880556Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:49:20.814259Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:49:42.989794Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:50:02.827925Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:50:15.606114Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:50:35.674932Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:50:46.445140Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:50:57.208311Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:51:08.052029Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T01:51:18.621438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T01:51:29.311Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 283 81 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "6cfd92fb-0c32-45ac-9a47-6c068218ab32" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:37934 10.133.0.169:443 10.132.0.14:53974 - v1-completions-path [2026-11-03T01:51:29.616Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 281 74 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "eb3107e0-839f-44ec-9a55-dd4e7aa43a39" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.169:37934 10.133.0.169:443 10.132.0.14:53988 - v1-completions-path [2026-11-03T01:51:30.196Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 43 - "34.228.250.194,10.132.0.14" "python-requests/2.34.2" "d536d595-48ea-402c-8a7d-9a80521e4e30" "maas.apps.5cd396bf50a6773ea167.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.169:443 10.132.0.14:53994 - v1-completions-path