2026-10-30T05:16:59.698453Z info FLAG: --concurrency="0" 2026-10-30T05:16:59.698473Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T05:16:59.698477Z info FLAG: --help="false" 2026-10-30T05:16:59.698479Z info FLAG: --log_as_json="false" 2026-10-30T05:16:59.698481Z info FLAG: --log_caller="" 2026-10-30T05:16:59.698483Z info FLAG: --log_output_level="default:info" 2026-10-30T05:16:59.698485Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T05:16:59.698489Z info FLAG: --log_target="[stdout]" 2026-10-30T05:16:59.698491Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T05:16:59.698493Z info FLAG: --outlierLogPath="" 2026-10-30T05:16:59.698495Z info FLAG: --profiling="true" 2026-10-30T05:16:59.698497Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T05:16:59.698499Z info FLAG: --proxyLogLevel="warning" 2026-10-30T05:16:59.698501Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T05:16:59.698503Z info FLAG: --stsPort="0" 2026-10-30T05:16:59.698505Z info FLAG: --templateFile="" 2026-10-30T05:16:59.698507Z info FLAG: --tokenManagerPlugin="" 2026-10-30T05:16:59.698509Z info FLAG: --vklog="0" 2026-10-30T05:16:59.698512Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T05:16:59.698673Z info Proxy role ips=[10.132.1.58] type=router id=maas-default-gateway-openshift-default-7f694df8bc-smfn4.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T05:16:59.698727Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T05:16:59.699932Z info cpu limit detected as 2, setting concurrency 2026-10-30T05:16:59.700162Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T05:16:59.700173Z info JWT policy is third-party-jwt 2026-10-30T05:16:59.700176Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T05:16:59.702701Z info Starting default Istio SDS Server 2026-10-30T05:16:59.702731Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T05:16:59.702752Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T05:16:59.702377Z info Opening status port 15020 2026-10-30T05:16:59.703673Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T05:16:59.703912Z info sds Starting SDS grpc server 2026-10-30T05:16:59.703929Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T05:16:59.705305Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T05:16:59.706208Z info Starting proxy agent 2026-10-30T05:16:59.706243Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T05:16:59.767744Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T05:16:59.877942Z info ads ADS: new connection for node:1 2026-10-30T05:16:59.878620Z info ads ADS: new connection for node:2 2026-10-30T05:16:59.883167Z info cache generated new workload certificate resourceName=default latency=179.158892ms ttl=23h59m59.116835774s 2026-10-30T05:16:59.883197Z info cache Root cert has changed, start rotating root cert 2026-10-30T05:16:59.883232Z info cache returned workload trust anchor from cache ttl=23h59m59.116768649s 2026-10-30T05:16:59.883249Z info cache returned workload certificate from cache ttl=23h59m59.116751731s 2026-10-30T05:16:59.883340Z info cache returned workload trust anchor from cache ttl=23h59m59.11666253s 2026-10-30T05:16:59.883551Z info cache returned workload trust anchor from cache ttl=23h59m59.116449383s 2026-10-30T05:16:59.964582Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:17:07.762907Z info Readiness succeeded in 8.067256526s 2026-10-30T05:17:07.763316Z info Envoy proxy is ready 2026-10-30T05:17:23.794927Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:17:26.827Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 147 412 100 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "302fcafe-8f28-4f29-8f0f-817f626eb4b4" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.30:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:49764 10.132.1.58:443 10.132.0.13:60684 - v1-chat-completions-path 2026-10-30T05:17:39.823031Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:17:42.375448Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:17:48.710818Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:17:55.978095Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:18:05.759794Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:18:08.465597Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:18:08.465635Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:18:08.465650Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T05:18:08.465655Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T05:18:08.175Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 315 290 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "6b2b6d9f-5538-4a5f-abfc-2e2483b3defc" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.58:43044 10.132.1.58:443 10.132.0.13:48594 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:18:20.546435Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:18:31.451769Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:18:33.471869Z info ads ADS: new connection for node:3 2026-10-30T05:18:33.472025Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T05:18:33.472051Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T05:18:35.537380Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:19:15.839894Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:19:16.688784Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:19:16.688834Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:19:16.688850Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:19:16.688856Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T05:19:16.574Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 313 114 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "a68b5cbd-6ac1-4615-8a49-611bd19aecf8" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.58:45650 10.132.1.58:443 10.132.0.13:33368 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:19:34.956959Z info ads ADS: "" 3 terminated 2026-10-30T05:19:35.001692Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:19:38.369162Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:19:46.203607Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:21:19.991647Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:21:37.161744Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:21:57.290425Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:11.570135Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:16.475536Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:20.363796Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:30.668614Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:35.467738Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:22:38.365Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 70 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "5b821da0-007a-474c-b9e6-7449a08ebd7c" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:43138 10.132.1.58:443 10.132.0.13:52958 - v1-completions-path 2026-10-30T05:22:50.847548Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:22:53.783601Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-30T05:22:53.791028Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:22:53.791070Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:22:53.791127Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:22:53.791136Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T05:22:53.713Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 330 77 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "4197614b-f009-43ed-8207-9a3d70f1637e" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.58:44164 10.132.1.58:443 10.132.0.13:36484 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:23:38.363713Z info ads ADS: new connection for node:4 2026-10-30T05:23:38.371280Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T05:23:38.371341Z info cache read certificate from file resource=file-root:system 2026-10-30T05:23:39.451922Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:23:41.501994Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:23:42.055559Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"129","StreamId":"2054024317873626537"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-10-30T05:23:42.035Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 20 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "dbe5572d-b3bb-48ce-b153-6595068abb95" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.1.58:443 10.132.0.13:48584 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T05:24:03.616417Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:24:03.616472Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:24:03.616483Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:24:03.616487Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T05:24:03.616651Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T05:24:03.509Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1716 106 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "237a67ec-da30-48e9-a8bd-44593fc384dc" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.58:56564 10.132.1.58:443 10.132.0.13:54148 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T05:24:18.662782Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:18.662815Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:18.662823Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:18.662828Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T05:24:18.662961Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T05:24:18.544Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1725 118 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "7a2d76e1-333c-40a9-a332-147e06757743" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.58:48134 10.132.1.58:443 10.132.0.13:51748 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T05:24:25.145348Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:24:25.145382Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:24:25.145397Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T05:24:25.145402Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:24:25.145552Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T05:24:25.053Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1724 92 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "e7072528-8157-40b3-bfb1-128c33f159a1" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.58:56564 10.132.1.58:443 10.132.0.13:51766 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T05:24:59.210744Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:59.210778Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:59.210789Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T05:24:59.210794Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:24:59.210943Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T05:24:59.112Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1716 98 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "b9f66c28-3bde-4963-803d-315a60200d67" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.1.58:48134 10.132.1.58:443 10.132.0.13:35208 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T05:25:16.986323Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:25:17.124248Z info ads ADS: "" 4 terminated 2026-10-30T05:25:17.372584Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:25:17.372621Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:25:17.372636Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T05:25:17.372641Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:25:17.425054Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T05:25:17.425092Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T05:25:17.425095Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T05:25:17.425098Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-10-30T05:25:17.296Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 74 316 76 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "098cb3ea-e4a0-478c-92d7-a0f60b72655f" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.58:34972 10.132.1.58:443 10.132.0.13:60230 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T05:25:17.392Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 32 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "2ac4452d-e622-4159-9468-d0fdc59be6d7" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.58:443 10.132.0.13:60232 - v1-completions-path 2026-10-30T05:25:18.576982Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:25:25.694188Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:25:41.477937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:25:46.162498Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:25:49.867517Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:25:49.867553Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:25:49.867567Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T05:25:49.867572Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T05:25:49.806Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 170 61 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "b4a1ace9-80a9-41a2-abe3-2bb04266ff9b" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:33664 10.132.1.58:443 10.132.0.13:50320 - v1-catch-all-path 2026-10-30T05:26:01.865728Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:26:17.805539Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:26:24.949Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 61 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "8c891210-d925-4031-88cb-9f565ef537a6" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:43138 10.132.1.58:443 10.132.0.13:48160 - v1-completions-path [2026-10-30T05:26:25.339Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 54 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "afcec197-76d1-41b3-b5f6-8a6d9bba7e20" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:43138 10.132.1.58:443 10.132.0.13:40530 - v1-completions-path 2026-10-30T05:26:41.457600Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:26:51.793617Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:26:56.437906Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:27:00.706Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 63 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "7540cd55-6cc3-46ee-9898-074857f96d9a" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:43138 10.132.1.58:443 10.132.0.13:42044 - v1-completions-path 2026-10-30T05:27:13.394880Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:15.908969Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:23.375626Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:33.003040Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:38.436326Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:46.628337Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:27:51.855954Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:10.959841Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:13.977857Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:29.501696Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:32.364589Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:41.504798Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:28:47.128112Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:28:47.128161Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:28:47.128182Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T05:28:47.128190Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T05:28:47.053Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 65 308 74 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "8635a750-1228-4a42-abc8-1af72f4b537d" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.132.0.56:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.1.58:54416 10.132.1.58:443 10.132.0.13:41702 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T05:28:59.517485Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:29:09.896956Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:29:26.572286Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:29:30.102902Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:31:01.941140Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:31:05.621213Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:32:31.629035Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:32:46.759018Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:32:49.313833Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:32:56.741062Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:33:13.626201Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:33:16.725166Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:33:17.072923Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-10-30T05:33:16.983Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 96 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "e30c9b8a-a424-4084-a828-77238d7b17c0" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:33664 10.132.1.58:443 10.132.0.13:46348 - v1-completions-path 2026-10-30T05:33:32.259843Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:33:43.495830Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:33:48.293448Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:05.307741Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:10.071471Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:13.306603Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:34:20.437Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 52 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "f21cb438-3247-4efc-88b5-5d6ab1f25d83" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.1.58:443 10.132.0.13:50222 - v1-completions-path [2026-10-30T05:34:21.660Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "8e5e89d5-2e5a-4925-86b0-571194fd7bb9" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:43138 10.132.1.58:443 10.132.0.13:50236 - v1-completions-path [2026-10-30T05:34:23.477Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 58 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "3680fb07-ee70-41f2-908c-59712fbce797" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:33664 10.132.1.58:443 10.132.0.13:50260 - v1-completions-path 2026-10-30T05:34:36.328335Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:46.672820Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:51.537472Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:34:54.604134Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:34:57.569Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 62 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "0330991d-6f4c-45ef-b93b-70bf2e3934fa" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:33664 10.132.1.58:443 10.132.0.13:42098 - v1-completions-path 2026-10-30T05:34:59.673644Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:35:04.495Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 59 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "e4e1090e-de03-44cc-9744-d9a7dd5a9144" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.134.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:43138 10.132.1.58:443 10.132.0.13:42142 - v1-completions-path 2026-10-30T05:35:18.033225Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:35:28.386203Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:35:30.858914Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:35:40.646487Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:35:45.244002Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:36:02.061897Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:36:22.027690Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:36:32.378069Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:36:42.843582Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:36:53.341000Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:03.698898Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:14.130152Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:24.427158Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:34.776566Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:45.125492Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T05:37:55.513172Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T05:38:02.461Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 100 - "52.71.36.72,10.132.0.13" "python-requests/2.34.2" "8d5e4f1e-6af7-4104-a458-c8d2e8e10ad3" "maas.apps.7e28d4d3eca2f6e06096.prod.konflux-ocp-ci.dev" "10.133.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.1.58:37994 10.132.1.58:443 10.132.0.13:59992 - v1-completions-path