2026-11-03T10:01:03.419566Z info FLAG: --concurrency="0" 2026-11-03T10:01:03.419588Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T10:01:03.419592Z info FLAG: --help="false" 2026-11-03T10:01:03.419594Z info FLAG: --log_as_json="false" 2026-11-03T10:01:03.419596Z info FLAG: --log_caller="" 2026-11-03T10:01:03.419598Z info FLAG: --log_output_level="default:info" 2026-11-03T10:01:03.419600Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T10:01:03.419604Z info FLAG: --log_target="[stdout]" 2026-11-03T10:01:03.419607Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T10:01:03.419608Z info FLAG: --outlierLogPath="" 2026-11-03T10:01:03.419610Z info FLAG: --profiling="true" 2026-11-03T10:01:03.419612Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T10:01:03.419614Z info FLAG: --proxyLogLevel="warning" 2026-11-03T10:01:03.419616Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T10:01:03.419618Z info FLAG: --stsPort="0" 2026-11-03T10:01:03.419620Z info FLAG: --templateFile="" 2026-11-03T10:01:03.419622Z info FLAG: --tokenManagerPlugin="" 2026-11-03T10:01:03.419624Z info FLAG: --vklog="0" 2026-11-03T10:01:03.419626Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T10:01:03.419812Z info Proxy role ips=[10.133.0.252] type=router id=maas-default-gateway-openshift-default-c9ffffc7f-c7mlq.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T10:01:03.419860Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T10:01:03.421070Z info cpu limit detected as 2, setting concurrency 2026-11-03T10:01:03.421309Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T10:01:03.421319Z info JWT policy is third-party-jwt 2026-11-03T10:01:03.421322Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T10:01:03.423004Z info Starting default Istio SDS Server 2026-11-03T10:01:03.423030Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T10:01:03.423062Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T10:01:03.423504Z info Opening status port 15020 2026-11-03T10:01:03.424251Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T10:01:03.425172Z info sds Starting SDS grpc server 2026-11-03T10:01:03.425206Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T10:01:03.425832Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T10:01:03.426764Z info Starting proxy agent 2026-11-03T10:01:03.426800Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T10:01:03.493476Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T10:01:03.637989Z info cache generated new workload certificate resourceName=default latency=212.63934ms ttl=23h59m59.362014454s 2026-11-03T10:01:03.638053Z info cache Root cert has changed, start rotating root cert 2026-11-03T10:01:03.638101Z info cache returned workload trust anchor from cache ttl=23h59m59.361899701s 2026-11-03T10:01:03.638821Z info ads ADS: new connection for node:1 2026-11-03T10:01:03.638866Z info cache returned workload certificate from cache ttl=23h59m59.36113514s 2026-11-03T10:01:03.640292Z info ads ADS: new connection for node:2 2026-11-03T10:01:03.640384Z info cache returned workload trust anchor from cache ttl=23h59m59.359617688s 2026-11-03T10:01:03.722489Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:01:11.633976Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:01:12.572090Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-03T10:01:12.921699Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-03T10:01:13.718759Z info Readiness succeeded in 10.302060941s 2026-11-03T10:01:13.719198Z info Envoy proxy is ready 2026-11-03T10:01:18.938686Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:01:28.550750Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:01:45.227901Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:01:57.023615Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:01:59.596938Z info ads ADS: new connection for node:3 2026-11-03T10:01:59.597130Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T10:01:59.597158Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T10:02:03.802719Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:02:52.774188Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T10:03:09.088Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 333 97 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "ff04699d-d094-498f-88ed-e2846c9ddd03" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.252:8000" outbound|54321||e2e-prio-pool-8906966c-inference-pool-ip-dc689d9c.llm.svc.cluster.local 10.133.0.252:60726 10.133.0.252:443 10.132.0.15:51908 - v1-model-routing 2026-11-03T10:03:26.899614Z info ads ADS: "" 3 terminated with stream closed 2026-11-03T10:03:27.200477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:03:32.009789Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:03:48.942024Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:06:47.589097Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:06:47.589136Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:06:47.589150Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:06:47.589155Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T10:06:47.454Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 33 325 134 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "10fd9015-8224-4cb5-bd80-9c97e1e3b505" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.252:37770 10.133.0.252:443 10.132.0.15:46780 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T10:07:09.424674Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:07:41.324473Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:08:10.712283Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:08:26.366493Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:08:43.243750Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:08:55.711021Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:09:00.912973Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:09:20.437719Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T10:10:02.001Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 78 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "458b0ad3-0216-4a4c-a877-5dff26a86592" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:39628 10.133.0.252:443 10.132.0.15:52234 - v1-completions-path 2026-11-03T10:10:18.471566Z info ads ADS: new connection for node:4 2026-11-03T10:10:18.478237Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T10:10:18.478303Z info cache read certificate from file resource=file-root:system 2026-11-03T10:10:21.943404Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:10:24.599003Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:10:36.683799Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"179","StreamId":"3128749684552675012"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-03T10:10:36.624Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 60 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "c7e69861-5a3f-4a1d-ba81-32c2871ae7eb" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.252:443 10.132.0.15:57996 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T10:11:41.609792Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:11:41.609822Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:11:41.609831Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:11:41.609835Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T10:11:41.610010Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T10:11:41.486Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 123 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "7967af31-6785-4929-9277-7992c07519e5" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.252:45798 10.133.0.252:443 10.132.0.15:57968 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T10:12:17.652447Z info ads ADS: "" 4 terminated 2026-11-03T10:12:17.695601Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:12:20.451301Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:12:31.133836Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:12:31.758662Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T10:12:31.758709Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T10:12:31.758729Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T10:12:31.758737Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T10:12:31.606Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 87 329 152 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "efcb2f4e-5c92-4e1d-8c13-c4e3b610dc7a" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.252:35524 10.133.0.252:443 10.132.0.15:56140 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T10:12:35.809Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 53 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "d89fad92-362b-45b7-b750-94ce528df798" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.252:443 10.132.0.15:56164 - v1-completions-path 2026-11-03T10:12:49.565271Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:12:54.274066Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:12:59.993425Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-03T10:12:59.905Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 88 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "fcfecc1b-1dc5-4ec1-b5e2-eac88dc1ab91" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:54946 10.133.0.252:443 10.132.0.15:34178 - v1-completions-path 2026-11-03T10:13:15.038878Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T10:13:23.400Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 70 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "22cc6a0f-5209-421a-8869-616bc9007dc3" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:53926 10.133.0.252:443 10.132.0.15:55850 - v1-completions-path 2026-11-03T10:13:41.824449Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:14:22.618023Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:14:33.359303Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:14:40.229734Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T10:14:56.260Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 57 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "035825a8-b82e-4904-a409-8bbc5a232ed7" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:54946 10.133.0.252:443 10.132.0.15:51042 - v1-completions-path 2026-11-03T10:14:57.339548Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-03T10:14:57.281Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 59 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "a4fe00c8-1eb6-4e1a-a6d4-9b839aeb14fc" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:40352 10.133.0.252:443 10.132.0.15:51050 - v1-completions-path 2026-11-03T10:15:11.283298Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:15:16.023098Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:15:28.174765Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:15:38.531111Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:16:05.114824Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:16:15.962617Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:16:33.973272Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T10:16:55.092Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 33 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "f7b76883-4325-4790-8704-7366155b6c00" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.252:443 10.132.0.15:40798 - v1-completions-path 2026-11-03T10:17:03.702720Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:17:25.374728Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:17:36.068729Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:21:42.948619Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:22:00.420550Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:22:05.148395Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T10:22:07.992Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 77 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "0015d826-2f4c-4e06-b17e-7fbc7cd5d500" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:39628 10.133.0.252:443 10.132.0.15:44632 - v1-completions-path 2026-11-03T10:22:18.674666Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:22:38.333899Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:22:43.305908Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:22:45.930510Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:22:45.930546Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:22:45.930560Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T10:22:45.930564Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:22:46.242326Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:22:46.242362Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T10:22:46.242378Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T10:22:46.242382Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T10:22:45.720Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 314 210 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "c5b2f297-0fa6-4427-99d9-fd0c506f5905" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.252:41696 10.133.0.252:443 10.132.0.15:40452 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T10:22:46.137Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 314 104 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "b89d1775-0332-4c2e-b12b-90a3a6ddf014" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.252:41696 10.133.0.252:443 10.132.0.15:40482 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T10:22:46.439Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 66 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "d7bd3503-9a7e-4739-b4da-589de63aaed8" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:54946 10.133.0.252:443 10.132.0.15:40492 - v1-completions-path 2026-11-03T10:23:03.286929Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:23:13.937861Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:23:20.382992Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T10:23:33.141Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 62 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "4dc90d34-e174-4adb-864b-1c412b7cd4f8" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:54946 10.133.0.252:443 10.132.0.15:49582 - v1-completions-path [2026-11-03T10:23:33.768Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 58 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "c80df48a-3154-41ed-ba02-cc72f194a70b" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:40352 10.133.0.252:443 10.132.0.15:49586 - v1-completions-path 2026-11-03T10:23:49.364843Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:23:55.877709Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:24:01.857102Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T10:24:12.616Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 56 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "35d0f626-02c9-498f-b9dd-23445122c8a0" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.252:443 10.132.0.15:48264 - v1-completions-path [2026-11-03T10:24:16.835Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 63 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "261c8d71-e45c-4581-a0c3-271146a585ab" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:54946 10.133.0.252:443 10.132.0.15:48282 - v1-completions-path 2026-11-03T10:24:20.455113Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T10:24:20.455152Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T10:24:20.455166Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T10:24:20.455171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T10:24:20.317Z] "DELETE /v1/api-keys/e2ca0ea9-a574-4ca4-b425-0fcde5ccc6f2 HTTP/1.1" 200 - via_upstream - "-" 0 365 137 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "ecb87830-ac8a-402d-bac5-715c10f9b5b1" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.252:60856 10.133.0.252:443 10.132.0.15:48288 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T10:24:34.266887Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:24:44.832292Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:24:51.481136Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:24:57.522760Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:25:09.400866Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T10:25:15.869Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 67 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "5aef73c7-e510-4972-a87e-bc15594486f0" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.54:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:40352 10.133.0.252:443 10.132.0.15:41278 - v1-completions-path 2026-11-03T10:25:33.101305Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:25:43.621398Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:25:48.493659Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:26:06.819977Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:26:12.999058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:26:36.608193Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:26:59.064603Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:27:20.670883Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:27:40.717316Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:27:51.507108Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:28:02.198492Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:28:12.996730Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:28:23.809927Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T10:28:33.786054Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T10:28:33.786090Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T10:28:33.786105Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T10:28:33.786110Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T10:28:34.409016Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T10:28:33.667Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 330 118 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "5cc3462c-a637-42f1-bc4f-656f3675da52" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.134.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.252:41746 10.133.0.252:443 10.132.0.15:41250 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T10:28:40.339Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 281 148 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "569e063e-0b03-400e-918a-87be9f498881" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:53926 10.133.0.252:443 10.132.0.15:41264 - v1-completions-path [2026-11-03T10:28:40.736Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 295 77 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "b90722ec-9ab4-4652-8cea-40992e8da18e" "maas.apps.7fee51e1a8dedf3477d2.prod.konflux-ocp-ci.dev" "10.132.0.55:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.252:39628 10.133.0.252:443 10.132.0.15:41266 - v1-completions-path