2026-11-02T15:45:33.763518Z info FLAG: --concurrency="0" 2026-11-02T15:45:33.763728Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T15:45:33.763766Z info FLAG: --help="false" 2026-11-02T15:45:33.763792Z info FLAG: --log_as_json="false" 2026-11-02T15:45:33.763816Z info FLAG: --log_caller="" 2026-11-02T15:45:33.763839Z info FLAG: --log_output_level="default:info" 2026-11-02T15:45:33.763896Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T15:45:33.763928Z info FLAG: --log_target="[stdout]" 2026-11-02T15:45:33.763952Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T15:45:33.763976Z info FLAG: --outlierLogPath="" 2026-11-02T15:45:33.764005Z info FLAG: --profiling="true" 2026-11-02T15:45:33.764029Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T15:45:33.764052Z info FLAG: --proxyLogLevel="warning" 2026-11-02T15:45:33.764074Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T15:45:33.764096Z info FLAG: --stsPort="0" 2026-11-02T15:45:33.764124Z info FLAG: --templateFile="" 2026-11-02T15:45:33.764146Z info FLAG: --tokenManagerPlugin="" 2026-11-02T15:45:33.764169Z info FLAG: --vklog="0" 2026-11-02T15:45:33.764191Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T15:45:33.764409Z info Proxy role ips=[10.132.0.197] type=router id=maas-default-gateway-openshift-default-77bfb58f8c-n9ld7.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T15:45:33.764527Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T15:45:33.767594Z info cpu limit detected as 2, setting concurrency 2026-11-02T15:45:33.768637Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T15:45:33.768710Z info JWT policy is third-party-jwt 2026-11-02T15:45:33.768739Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T15:45:33.770819Z info Starting default Istio SDS Server 2026-11-02T15:45:33.770998Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T15:45:33.771055Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T15:45:33.770901Z info Opening status port 15020 2026-11-02T15:45:33.774377Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T15:45:33.793470Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T15:45:33.795518Z info Starting proxy agent 2026-11-02T15:45:33.795625Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T15:45:33.798786Z info sds Starting SDS grpc server 2026-11-02T15:45:33.798818Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T15:45:33.921252Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T15:45:34.150157Z info ads ADS: new connection for node:1 2026-11-02T15:45:34.152042Z info ads ADS: new connection for node:2 2026-11-02T15:45:34.152924Z info ads ADS: new connection for node:3 2026-11-02T15:45:34.153120Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T15:45:34.153151Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T15:45:34.245919Z info cache generated new workload certificate resourceName=default latency=467.261777ms ttl=23h59m59.75408494s 2026-11-02T15:45:34.246032Z info cache Root cert has changed, start rotating root cert 2026-11-02T15:45:34.246126Z info cache returned workload trust anchor from cache ttl=23h59m59.753875262s 2026-11-02T15:45:34.246193Z info cache returned workload certificate from cache ttl=23h59m59.753808339s 2026-11-02T15:45:34.246356Z info cache returned workload trust anchor from cache ttl=23h59m59.753644852s 2026-11-02T15:45:34.246483Z info cache returned workload trust anchor from cache ttl=23h59m59.753517708s 2026-11-02T15:45:34.388129Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:45:42.816415Z info Readiness succeeded in 9.056063025s 2026-11-02T15:45:42.816832Z info Envoy proxy is ready 2026-11-02T15:46:08.434160Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:46:11.581494Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T15:46:11.581534Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T15:46:11.581547Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T15:46:11.581552Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-02T15:46:11.449Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 312 132 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "618c8468-3bef-4677-8e33-ac3e567193f8" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.197:54248 10.132.0.197:443 10.132.0.22:36082 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T15:46:14.866Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 356 89 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "3114fb54-138e-4227-a6ec-deb725694eeb" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.196:8000" outbound|54321||e2e-prio-pool-b23739ca-inference-pool-ip-d5244b36.llm.svc.cluster.local 10.132.0.197:49584 10.132.0.197:443 10.132.0.22:46110 - v1-model-routing [2026-11-02T15:46:30.595Z] "POST /v1/chat/completions HTTP/1.1" 403 - - "-" 131 0 34 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "dc914fec-536a-44f7-bd1b-65b5b9e6a4f6" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "-" outbound|54321||e2e-prio-pool-b23739ca-inference-pool-ip-d5244b36.llm.svc.cluster.local - 10.132.0.197:443 10.132.0.22:50676 - v1-model-routing 2026-11-02T15:46:32.740181Z info ads ADS: "" 3 terminated 2026-11-02T15:46:33.031140Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:46:37.930950Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:46:55.134365Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:49:28.946790Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:49:58.501474Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:50:28.260533Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:50:42.252520Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:50:51.362688Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:51:03.278104Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:51:08.317124Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:51:26.374741Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:51:28.566450Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:51:28.566487Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:51:28.566501Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:51:28.566506Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T15:51:28.453Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 331 113 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "bd590e7a-b71a-4a8b-92af-1a4a70526ffe" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.197:36934 10.132.0.197:443 10.132.0.22:43064 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T15:51:29.363Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 292 75 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "3b34a7b2-a924-48e3-8240-94aa3f6f0483" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:56016 10.132.0.197:443 10.132.0.22:43072 - v1-completions-path 2026-11-02T15:51:43.260307Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-02T15:51:43.196Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 288 65 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "1519c951-9021-497a-b00a-7690cd90acbb" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:56016 10.132.0.197:443 10.132.0.22:45932 - v1-completions-path [2026-11-02T15:52:13.938Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 73 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "f174615f-bd26-453e-8bfd-7500de8b5dc4" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:34816 10.132.0.197:443 10.132.0.22:38800 - v1-completions-path 2026-11-02T15:52:15.676678Z info ads ADS: new connection for node:4 2026-11-02T15:52:15.681440Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T15:52:15.681497Z info cache read certificate from file resource=file-root:system 2026-11-02T15:52:19.589347Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:52:20.808601Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:52:24.041209Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"138","StreamId":"15156067433092322522"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=26 [2026-11-02T15:52:23.981Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 59 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "4552e906-0229-4f60-bbeb-366a9b9085d4" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.0.197:443 10.132.0.22:33412 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-02T15:52:26.311997Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"141","StreamId":"12403248959848746069"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-02T15:52:26.243Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 68 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "6a0aca3c-37a3-4540-963e-d968ee562ca0" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.0.197:443 10.132.0.22:33444 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-02T15:52:44.843229Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:44.843270Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:44.843285Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:44.843291Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T15:52:44.843534Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T15:52:44.720Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1726 123 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "d71356ff-d884-4059-bd57-ff053358099a" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.197:56960 10.132.0.197:443 10.132.0.22:46376 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T15:52:47.147891Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:47.147920Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:47.147930Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T15:52:47.147934Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:47.148086Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T15:52:47.045Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1725 102 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "31c2d7d8-2a02-43fa-a289-acc2311a26e4" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.197:56960 10.132.0.197:443 10.132.0.22:46404 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T15:53:40.674754Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:40.674786Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:40.674796Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:40.674800Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T15:53:40.674980Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T15:53:40.563Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 111 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "145c2848-91c5-43fc-9c6f-5f286a954175" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.197:56960 10.132.0.197:443 10.132.0.22:44436 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T15:53:42.985209Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:42.985244Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:42.985255Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T15:53:42.985259Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:42.985418Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T15:53:42.878Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1724 107 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "c8aae26a-1393-46b8-bfa1-175a145424b2" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.197:56960 10.132.0.197:443 10.132.0.22:44458 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T15:53:58.267490Z info ads ADS: "" 4 terminated 2026-11-02T15:53:58.308768Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:54:01.129143Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:54:11.852669Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:54:30.525108Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:54:35.249790Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:54:38.003Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 298 80 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "341ec28e-91ce-4b9f-bdbd-90a41bf7ab9c" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:37920 10.132.0.197:443 10.132.0.22:47498 - v1-completions-path [2026-11-02T15:54:38.563Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 58 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "43d788ff-11ec-48b6-b1f2-b2655a8fe219" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.197:443 10.132.0.22:47510 - v1-completions-path 2026-11-02T15:54:56.599010Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:55:09.593790Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:55:16.716Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 288 71 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "03550b22-b940-4895-9fb3-e89c3f5d7282" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:37920 10.132.0.197:443 10.132.0.22:58246 - v1-completions-path 2026-11-02T15:55:18.165778Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 [2026-11-02T15:55:18.103Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 63 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "ffdca297-e621-45d1-85f9-36df04528e99" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:35082 10.132.0.197:443 10.132.0.22:58272 - v1-completions-path [2026-11-02T15:55:19.769Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 58 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "1f92d771-46b6-485d-9bd8-a70ea14e3963" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:37920 10.132.0.197:443 10.132.0.22:58284 - v1-completions-path 2026-11-02T15:55:40.568810Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:55:51.182722Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:55:57.764182Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:56:08.642Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 62 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "9cbde0ac-5a53-4ddf-bf4d-01c51eab3f59" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:35082 10.132.0.197:443 10.132.0.22:37366 - v1-completions-path [2026-11-02T15:56:10.494Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 62 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "f1126bb9-2488-45ea-a3b0-899cb79f018e" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:35082 10.132.0.197:443 10.132.0.22:37380 - v1-completions-path [2026-11-02T15:56:11.525Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 60 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "d135bcd7-0849-45b2-924e-51895d03aeaa" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:37920 10.132.0.197:443 10.132.0.22:37386 - v1-completions-path [2026-11-02T15:56:12.047Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "1ffc5d9c-7057-460a-81d5-392be62ea030" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:35082 10.132.0.197:443 10.132.0.22:37390 - v1-completions-path 2026-11-02T15:56:25.991914Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:56:30.810488Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:56:44.401595Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:56:53.818716Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:57:03.410302Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:57:06.070Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 295 66 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "2bbf0bab-f48d-4dcb-921b-83812296fdc9" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:56016 10.132.0.197:443 10.132.0.22:60184 - v1-completions-path 2026-11-02T15:57:12.797174Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:57:29.370083Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:57:42.463142Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:58:13.881358Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:58:36.736211Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:58:47.410131Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:03:11.573073Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:03:29.090219Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:03:33.730851Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:03:47.178437Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:02.969151Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:08.103166Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:28.184514Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:38.896472Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:45.241598Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:57.322917Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T16:04:57.322953Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T16:04:57.322970Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-02T16:04:57.322975Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-11-02T16:04:57.187Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 309 135 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "7d7b0c1e-9778-49ab-8da5-2e337d93403f" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.208:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.197:49858 10.132.0.197:443 10.132.0.22:41594 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T16:04:57.834Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 65 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "ab73180b-72da-4491-bf72-8745016d1051" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:37920 10.132.0.197:443 10.132.0.22:41596 - v1-completions-path 2026-11-02T16:05:14.510203Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:05:20.963612Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:05:27.248902Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T16:05:36.934Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 62 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "6861d1db-899d-443e-b7d0-9fed0100320e" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:37920 10.132.0.197:443 10.132.0.22:60066 - v1-completions-path [2026-11-02T16:05:41.198Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 62 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "f877326e-448a-46f2-bd99-33e92d1cfaf1" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:37920 10.132.0.197:443 10.132.0.22:60096 - v1-completions-path [2026-11-02T16:05:44.085Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 64 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "b39f44bf-91df-473a-ab21-e7088fe82841" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.197:37920 10.132.0.197:443 10.132.0.22:41318 - v1-completions-path 2026-11-02T16:05:59.217219Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:06:09.834706Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:06:16.619093Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:06:22.778445Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:06:34.818749Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:06:58.065835Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:07:08.857731Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:07:13.854983Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:07:32.349576Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:07:38.592022Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:08:01.900894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:08:23.958147Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:08:43.750021Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:08:54.763134Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:05.643648Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:16.531054Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:27.173675Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:37.980491Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:48.795732Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:59.485219Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:10:05.228460Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-02T16:10:05.228486Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-02T16:10:05.228489Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-11-02T16:10:05.228493Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=26 [2026-11-02T16:10:05.205Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 22 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "3f6bdca6-1e97-425d-a3a8-0736d401c3ef" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.197:443 10.132.0.22:57490 - v1-completions-path