2026-11-02T18:55:08.947736Z info FLAG: --concurrency="0" 2026-11-02T18:55:08.947757Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T18:55:08.947761Z info FLAG: --help="false" 2026-11-02T18:55:08.947763Z info FLAG: --log_as_json="false" 2026-11-02T18:55:08.947765Z info FLAG: --log_caller="" 2026-11-02T18:55:08.947767Z info FLAG: --log_output_level="default:info" 2026-11-02T18:55:08.947769Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T18:55:08.947773Z info FLAG: --log_target="[stdout]" 2026-11-02T18:55:08.947775Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T18:55:08.947777Z info FLAG: --outlierLogPath="" 2026-11-02T18:55:08.947779Z info FLAG: --profiling="true" 2026-11-02T18:55:08.947781Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T18:55:08.947783Z info FLAG: --proxyLogLevel="warning" 2026-11-02T18:55:08.947785Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T18:55:08.947787Z info FLAG: --stsPort="0" 2026-11-02T18:55:08.947789Z info FLAG: --templateFile="" 2026-11-02T18:55:08.947791Z info FLAG: --tokenManagerPlugin="" 2026-11-02T18:55:08.947793Z info FLAG: --vklog="0" 2026-11-02T18:55:08.947796Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T18:55:08.947935Z info Proxy role ips=[10.132.0.206] type=router id=maas-default-gateway-openshift-default-9dccdc9b7-j8pgt.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T18:55:08.947979Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T18:55:08.949475Z info cpu limit detected as 2, setting concurrency 2026-11-02T18:55:08.949659Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T18:55:08.949670Z info JWT policy is third-party-jwt 2026-11-02T18:55:08.949673Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T18:55:08.951111Z info Starting default Istio SDS Server 2026-11-02T18:55:08.951157Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T18:55:08.951176Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T18:55:08.951527Z info Opening status port 15020 2026-11-02T18:55:08.952477Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T18:55:08.953809Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T18:55:08.954629Z info Starting proxy agent 2026-11-02T18:55:08.954660Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T18:55:08.963739Z info sds Starting SDS grpc server 2026-11-02T18:55:08.963760Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T18:55:09.018645Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T18:55:09.178215Z info ads ADS: new connection for node:1 2026-11-02T18:55:09.178931Z info ads ADS: new connection for node:2 2026-11-02T18:55:09.260761Z info cache generated new workload certificate resourceName=default latency=308.493027ms ttl=23h59m59.739243465s 2026-11-02T18:55:09.260798Z info cache Root cert has changed, start rotating root cert 2026-11-02T18:55:09.260841Z info cache returned workload trust anchor from cache ttl=23h59m59.739159597s 2026-11-02T18:55:09.260862Z info cache returned workload certificate from cache ttl=23h59m59.739138567s 2026-11-02T18:55:09.260960Z info cache returned workload trust anchor from cache ttl=23h59m59.739044282s 2026-11-02T18:55:09.261210Z info cache returned workload trust anchor from cache ttl=23h59m59.738790283s 2026-11-02T18:55:09.350423Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:17.446544Z info Readiness succeeded in 8.501082206s 2026-11-02T18:55:17.446841Z info Envoy proxy is ready 2026-11-02T18:55:29.107239Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:55:32.542Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 108118 108394 366 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "9a25c27b-4eb2-4a74-8c90-27c4b8534b10" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:48838 10.132.0.206:443 10.132.0.19:45544 - v1-chat-completions-path [2026-11-02T18:55:34.560Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 140 405 60 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "545def1b-d2c9-4553-acc7-6e60ec38e254" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:48838 10.132.0.206:443 10.132.0.19:45550 - v1-chat-completions-path 2026-11-02T18:55:46.824565Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T18:55:46.824594Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T18:55:46.824598Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T18:55:46.824602Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-02T18:55:46.675Z] "POST /llm/e2e-embedding-simulated/v1/embeddings HTTP/1.1" 403 - - "-" 61 103 149 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "9dc6ce21-6a88-4f17-824f-3b4282d85da4" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.206:443 10.132.0.19:38186 - v1-catch-all-path 2026-11-02T18:55:48.133375Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:52.734375Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:55.285865Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:55:55.285902Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:55:55.285915Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:55:55.285920Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T18:55:54.996Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 289 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "806d604c-49a2-4fcf-81de-31d4c245c097" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.132.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.206:49682 10.132.0.206:443 10.132.0.19:38200 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T18:56:02.384776Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:15.561213Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:24.705687Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:41.458225Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:53.362787Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:56.213307Z info ads ADS: new connection for node:3 2026-11-02T18:56:56.213490Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:56:56.213520Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:56:59.991270Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:39.776457Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:58:04.144152Z info ads ADS: "" 3 terminated 2026-11-02T18:58:04.192351Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:58:09.040695Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:58:25.527617Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:00:58.616662Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:01:26.391837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:01:52.723909Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:02:07.649474Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:02:14.199616Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:02:26.006277Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:02:30.768598Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:02:49.113715Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T19:03:35.860Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 80 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "6199ec6d-043d-4d4a-a8f4-5b051cf702eb" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:56616 10.132.0.206:443 10.132.0.19:58492 - v1-completions-path 2026-11-02T19:03:37.611351Z info ads ADS: new connection for node:4 2026-11-02T19:03:37.617021Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T19:03:37.617067Z info cache read certificate from file resource=file-root:system 2026-11-02T19:03:41.554529Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:03:42.669629Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:03:45.844332Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-02T19:03:45.845181Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"161","StreamId":"10674991672394818650"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-02T19:03:45.789Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 56 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "b724222d-a4bd-4c02-8e6a-76b6a0446a76" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.0.206:443 10.132.0.19:41912 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-02T19:03:48.115749Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"164","StreamId":"13288795441464722539"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-02T19:03:48.047Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 68 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "c63f43e8-b265-4c99-a05e-feac8898ff3e" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.0.206:443 10.132.0.19:41924 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-02T19:04:48.359620Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T19:04:48.359661Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T19:04:48.359677Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T19:04:48.359683Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T19:04:48.359883Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T19:04:48.233Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1708 126 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "f061d87b-e0e1-4000-9ad3-7a2380810c75" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.206:41760 10.132.0.206:443 10.132.0.19:41804 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T19:05:26.833001Z info ads ADS: "" 4 terminated 2026-11-02T19:05:26.883920Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:05:29.404586Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:05:39.917292Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:05:58.243881Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:06:02.681782Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T19:06:06.988Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 75 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "76816abc-4a8c-4144-ba64-89d3bc9fee42" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:42262 10.132.0.206:443 10.132.0.19:37484 - v1-completions-path [2026-11-02T19:06:07.511Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 64 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "ed7e20c0-cb7f-4876-b00a-c19b0cbdff34" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:42266 10.132.0.206:443 10.132.0.19:37498 - v1-completions-path 2026-11-02T19:06:26.463328Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:06:39.849208Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T19:06:48.749Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 61 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "737569dc-b010-4820-a351-23278e3837eb" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.206:443 10.132.0.19:50070 - v1-completions-path 2026-11-02T19:07:10.708464Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:07:21.506554Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:07:27.687894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T19:07:41.351Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 62 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "e88fb9a9-2804-4d79-b920-7164c8f493a1" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:42262 10.132.0.206:443 10.132.0.19:37952 - v1-completions-path [2026-11-02T19:07:41.872Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 56 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "dd720ac2-5561-4602-b8a8-ecd2cb260363" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:42262 10.132.0.206:443 10.132.0.19:37958 - v1-completions-path [2026-11-02T19:07:42.642Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 56 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "5ba29f59-454f-49db-806a-d53f0ad5299c" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:42262 10.132.0.206:443 10.132.0.19:37962 - v1-completions-path 2026-11-02T19:07:56.279994Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:08:00.919992Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:08:14.019386Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:08:23.830847Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:08:33.039553Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:08:41.947455Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:08:49.525371Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:09:09.666952Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:09:15.686159Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:09:33.949407Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:09:39.366952Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:09:50.554513Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:10:17.686599Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:10:28.389920Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:10:34.027300Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T19:10:34.027337Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T19:10:34.027355Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T19:10:34.027360Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T19:10:33.913Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 113 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "3ab9b27d-3474-450c-874b-f824d3e897f0" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.132.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.206:44554 10.132.0.206:443 10.132.0.19:46148 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T19:10:49.454852Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:10:55.486801Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:13:47.275384Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:15:17.016937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:15:36.619345Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:15:41.639170Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:15:44.855775Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T19:15:44.855803Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T19:15:44.855806Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T19:15:44.855810Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T19:15:44.822Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 33 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "396e5014-945a-4b89-b707-bd984079ecc9" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.206:443 10.132.0.19:38448 - v1-completions-path 2026-11-02T19:15:54.680018Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:16:11.124256Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:16:16.058226Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:16:18.782526Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T19:16:18.782565Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T19:16:18.782579Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T19:16:18.782585Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T19:16:18.633Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 314 149 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "fe80cdee-1a85-4f89-a8d3-a49a9a29a88d" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.1.110:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.206:45944 10.132.0.206:443 10.132.0.19:56822 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T19:16:20.182Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 60 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "23a8d0cc-4886-4dab-8f21-b33a158e7ee4" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:42262 10.132.0.206:443 10.132.0.19:56826 - v1-completions-path 2026-11-02T19:16:22.271890Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T19:16:22.271928Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T19:16:22.271942Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T19:16:22.271947Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T19:16:22.166Z] "DELETE /v1/api-keys/2335da20-0473-4d7f-b02c-3b184df42595 HTTP/1.1" 200 - via_upstream - "-" 0 369 105 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "5a486dc8-0dc6-4c74-b53e-6dad8f17cfac" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.1.110:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.206:45944 10.132.0.206:443 10.132.0.19:56838 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T19:16:35.866320Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:16:46.452418Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:16:52.663326Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T19:17:03.714Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 63 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "960c88fc-46d6-4454-a714-48aabb927548" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:42262 10.132.0.206:443 10.132.0.19:39306 - v1-completions-path 2026-11-02T19:17:19.939049Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:17:26.186246Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:17:32.211767Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:17:40.702600Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T19:17:40.702636Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T19:17:40.702654Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T19:17:40.702659Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T19:17:40.576Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 66 308 126 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "0341dfc1-0508-445f-a3a5-f7cb0a970e77" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.132.0.225:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.206:47464 10.132.0.206:443 10.132.0.19:48176 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T19:17:51.311513Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T19:17:51.311552Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T19:17:51.311567Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T19:17:51.311571Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T19:17:51.151Z] "DELETE /v1/api-keys/66b4657d-52ed-4a1a-a131-87dce13e0f7e HTTP/1.1" 200 - via_upstream - "-" 0 365 160 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "14e130f3-5f73-4adc-8316-2a791e9312e2" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.132.0.225:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.206:49578 10.132.0.206:443 10.132.0.19:53386 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T19:18:04.761691Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:18:15.611410Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:18:22.352642Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:18:28.364257Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:18:37.609446Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:18:45.288581Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-02T19:18:45.282Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 63 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "2075ea67-3048-40dc-81af-523c9e02b22c" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:42266 10.132.0.206:443 10.132.0.19:56862 - v1-completions-path [2026-11-02T19:18:46.010Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 59 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "85eb21bd-1478-4c3c-bc3f-fee493db4160" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:42262 10.132.0.206:443 10.132.0.19:56882 - v1-completions-path [2026-11-02T19:18:46.372Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 60 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "da3eb098-30a4-40b4-94f9-addb7b0034bc" "maas.apps.ee7d99a15883b5bfd98c.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.206:42266 10.132.0.206:443 10.132.0.19:56888 - v1-completions-path 2026-11-02T19:19:00.615893Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:19:11.304149Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:19:16.063957Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:19:34.487937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:19:40.555881Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:20:04.490220Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:20:26.308251Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:20:46.396463Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:20:57.065655Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:21:07.861910Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:21:18.640420Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:21:29.311187Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:21:40.171783Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:21:50.903612Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T19:22:01.552271Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8