2026-11-02T15:45:33.832468Z info FLAG: --concurrency="0" 2026-11-02T15:45:33.832619Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T15:45:33.832643Z info FLAG: --help="false" 2026-11-02T15:45:33.832659Z info FLAG: --log_as_json="false" 2026-11-02T15:45:33.832674Z info FLAG: --log_caller="" 2026-11-02T15:45:33.832690Z info FLAG: --log_output_level="default:info" 2026-11-02T15:45:33.832705Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T15:45:33.832723Z info FLAG: --log_target="[stdout]" 2026-11-02T15:45:33.832739Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T15:45:33.832754Z info FLAG: --outlierLogPath="" 2026-11-02T15:45:33.832769Z info FLAG: --profiling="true" 2026-11-02T15:45:33.832784Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T15:45:33.832799Z info FLAG: --proxyLogLevel="warning" 2026-11-02T15:45:33.832814Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T15:45:33.832829Z info FLAG: --stsPort="0" 2026-11-02T15:45:33.832844Z info FLAG: --templateFile="" 2026-11-02T15:45:33.832881Z info FLAG: --tokenManagerPlugin="" 2026-11-02T15:45:33.832904Z info FLAG: --vklog="0" 2026-11-02T15:45:33.832911Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T15:45:33.833064Z info Proxy role ips=[10.132.0.198] type=router id=maas-default-gateway-openshift-default-77bfb58f8c-nb58d.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T15:45:33.833120Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T15:45:33.834224Z info cpu limit detected as 2, setting concurrency 2026-11-02T15:45:33.834689Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T15:45:33.834706Z info JWT policy is third-party-jwt 2026-11-02T15:45:33.834718Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T15:45:33.836372Z info Starting default Istio SDS Server 2026-11-02T15:45:33.836398Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T15:45:33.836431Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T15:45:33.836781Z info Opening status port 15020 2026-11-02T15:45:33.837665Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T15:45:33.838883Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T15:45:33.840073Z info Starting proxy agent 2026-11-02T15:45:33.840110Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T15:45:33.861910Z info sds Starting SDS grpc server 2026-11-02T15:45:33.862006Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T15:45:34.009290Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T15:45:34.101263Z info cache generated new workload certificate resourceName=default latency=261.497329ms ttl=23h59m59.898743844s 2026-11-02T15:45:34.101432Z info cache Root cert has changed, start rotating root cert 2026-11-02T15:45:34.101524Z info cache returned workload trust anchor from cache ttl=23h59m59.898477329s 2026-11-02T15:45:34.224128Z info ads ADS: new connection for node:1 2026-11-02T15:45:34.224204Z info cache returned workload certificate from cache ttl=23h59m59.775798249s 2026-11-02T15:45:34.224525Z info ads ADS: new connection for node:2 2026-11-02T15:45:34.224590Z info cache returned workload trust anchor from cache ttl=23h59m59.775411265s 2026-11-02T15:45:34.225164Z info ads ADS: new connection for node:3 2026-11-02T15:45:34.225542Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T15:45:34.225573Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T15:45:34.372614Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:45:42.845727Z info Readiness succeeded in 9.015756903s 2026-11-02T15:45:42.846121Z info Envoy proxy is ready 2026-11-02T15:46:08.418848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:46:32.737419Z info ads ADS: "" 3 terminated 2026-11-02T15:46:33.043504Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:46:37.944222Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:46:55.158052Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:49:28.945700Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:49:58.487235Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:50:28.216064Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:50:42.229024Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:50:51.362522Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:51:03.277697Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:51:08.315932Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:51:11.496Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 86 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "88dba90b-cbea-4ac6-bd5a-e8157cf49175" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.198:42972 10.132.0.198:443 10.132.0.22:39178 - v1-completions-path 2026-11-02T15:51:26.355740Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:51:44.037Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 288 83 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "163f1db5-27f0-4c29-9f71-711fbcf6ec19" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.198:57528 10.132.0.198:443 10.132.0.22:33524 - v1-completions-path 2026-11-02T15:52:15.672901Z info ads ADS: new connection for node:4 2026-11-02T15:52:15.680932Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T15:52:15.680992Z info cache read certificate from file resource=file-root:system 2026-11-02T15:52:19.616189Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:52:20.831081Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:52:23.781338Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T15:52:23.781376Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T15:52:23.781390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T15:52:23.781396Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T15:52:23.660Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 120 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "2d8174a3-1115-4071-90ab-17604bd6e79e" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.198:51606 10.132.0.198:443 10.132.0.22:37208 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T15:52:30.843241Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"133","StreamId":"1554190313949805772"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=25 [2026-11-02T15:52:30.780Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 63 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "595f18ab-ddfb-4f77-8d92-a455223aca79" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.132.0.198:443 10.132.0.22:41488 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-02T15:52:40.217459Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:40.217494Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:40.217507Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T15:52:40.217513Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:40.217719Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T15:52:40.096Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 120 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "802649aa-c5ee-4ddd-823c-1106c788aab7" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.198:50840 10.132.0.198:443 10.132.0.22:54296 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T15:52:42.471778Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-02T15:52:42.521617Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:42.521647Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:42.521656Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:42.521661Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T15:52:42.521807Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T15:52:42.415Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 106 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "46d1aac2-d6b2-43f9-8192-2afb41abeefa" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.198:50840 10.132.0.198:443 10.132.0.22:54308 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T15:52:56.454437Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:56.454475Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:56.454485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T15:52:56.454491Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:52:56.454647Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T15:52:56.346Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1723 108 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "ff1ee9e3-f83c-4949-b8d9-d776bfb2359a" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.198:50840 10.132.0.198:443 10.132.0.22:32784 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T15:53:15.077987Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:15.078022Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:15.078035Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T15:53:15.078041Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:15.078230Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T15:53:14.962Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 115 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "992307c9-049a-4470-8aee-ab3aa879c071" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.198:50840 10.132.0.198:443 10.132.0.22:54420 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T15:53:28.997295Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:28.997327Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:28.997336Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:53:28.997341Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T15:53:28.997491Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T15:53:28.883Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 114 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "d042264d-4f53-4f09-829c-069015c6bedc" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.198:50840 10.132.0.198:443 10.132.0.22:52172 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T15:53:44.884110Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T15:53:44.884147Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T15:53:44.884163Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T15:53:44.884168Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T15:53:44.770Z] "DELETE /v1/api-keys/7b9bf19f-3225-444d-b681-989df03a81fc HTTP/1.1" 200 - via_upstream - "-" 0 359 114 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "c1b1f029-d46b-4fd5-a81d-34928317831b" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.198:58462 10.132.0.198:443 10.132.0.22:49538 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T15:53:58.278335Z info ads ADS: "" 4 terminated with stream closed 2026-11-02T15:53:58.352389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:54:01.087264Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:54:11.830410Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:54:30.524621Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:54:35.247708Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:54:56.598291Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:55:09.595692Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:55:16.989Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 69 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "460fde4f-960b-494d-b63c-52f11a6181af" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.198:42178 10.132.0.198:443 10.132.0.22:56556 - v1-completions-path 2026-11-02T15:55:21.936223Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-02T15:55:21.879Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 57 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "98edb79f-de93-4128-b581-e968f00b0d70" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.198:42972 10.132.0.198:443 10.132.0.22:56576 - v1-completions-path 2026-11-02T15:55:40.530787Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:55:51.201002Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:55:57.793171Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:56:08.187219Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T15:56:08.187265Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T15:56:08.187286Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T15:56:08.187293Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T15:56:08.064Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 318 123 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "8993c9b2-7322-4904-92c8-8caed9fbc36b" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.198:47634 10.132.0.198:443 10.132.0.22:49344 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T15:56:10.756Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 60 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "db8106a4-4e90-4785-91a2-890a0636eba0" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.198:42178 10.132.0.198:443 10.132.0.22:49356 - v1-completions-path [2026-11-02T15:56:11.789Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 57 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "b2499dd8-c7e0-4051-b4d9-342c39665478" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.198:42972 10.132.0.198:443 10.132.0.22:49366 - v1-completions-path 2026-11-02T15:56:26.028585Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:56:30.829525Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:56:44.420662Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:56:53.822713Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:57:03.428571Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:57:12.795200Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:57:26.946Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 74 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "b68da4b4-8912-479f-8053-7ba1fcd8c56a" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.198:57528 10.132.0.198:443 10.132.0.22:49750 - v1-completions-path 2026-11-02T15:57:29.372577Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:57:42.488846Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:58:13.860365Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:58:36.731417Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:58:47.410329Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:03:11.543731Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:03:29.073584Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:03:33.749286Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:03:47.151093Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:02.939774Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:08.091092Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:14.112473Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T16:04:14.112512Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T16:04:14.112530Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T16:04:14.112534Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T16:04:13.995Z] "DELETE /v1/api-keys/b796348d-76c4-4413-b004-5357f93d026d HTTP/1.1" 200 - via_upstream - "-" 0 369 117 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "f9789520-a6e3-48d1-b883-93a4dbaa6cf0" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.208:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.198:57608 10.132.0.198:443 10.132.0.22:48554 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T16:04:28.173329Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:38.864906Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:04:45.230304Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:05:14.553660Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:05:21.002461Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:05:27.273291Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T16:05:37.659Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 59 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "77b9d423-174e-4ff2-a636-95c138620f6f" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.198:443 10.132.0.22:51704 - v1-completions-path [2026-11-02T16:05:44.449Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 63 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "98cab5a2-a3c1-4ae1-af32-4e6ea8b34675" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.198:42972 10.132.0.198:443 10.132.0.22:55932 - v1-completions-path 2026-11-02T16:05:59.230629Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:06:09.833233Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:06:16.635129Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:06:22.816438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:06:34.793102Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:06:58.065770Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:07:08.837041Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:07:13.855156Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:07:32.325938Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:07:38.590342Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:08:01.899515Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:08:23.932540Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:08:43.755757Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:08:54.757159Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:05.642376Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:16.505007Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:27.170200Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:37.984361Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:48.781741Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:09:59.490924Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T16:10:07.465093Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T16:10:07.465118Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T16:10:07.465121Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T16:10:07.465125Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T16:10:07.441Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 23 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "d58f7318-f778-46be-bf0a-67a59ec31355" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.198:443 10.132.0.22:35494 - v1-completions-path 2026-11-02T16:10:13.118063Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T16:10:13.118090Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T16:10:13.118093Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T16:10:13.118097Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-02T16:10:12.580Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 281 77 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "95badc8f-7598-42bb-9073-148e6ee28a8f" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "10.132.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.198:49472 10.132.0.198:443 10.132.0.22:35516 - v1-completions-path [2026-11-02T16:10:12.858Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 36 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "d2cb4eb6-b713-48a2-8edf-9afd26bd9d75" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.198:443 10.132.0.22:35524 - v1-completions-path [2026-11-02T16:10:13.091Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 26 - "34.228.250.194,10.132.0.22" "python-requests/2.34.2" "6b3c1f45-664d-41e8-90e0-657747fe30d2" "maas.apps.61d88bbb8c406d3a9034.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.198:443 10.132.0.22:35536 - v1-completions-path