2026-11-03T16:32:39.333513Z info FLAG: --concurrency="0" 2026-11-03T16:32:39.333534Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T16:32:39.333538Z info FLAG: --help="false" 2026-11-03T16:32:39.333540Z info FLAG: --log_as_json="false" 2026-11-03T16:32:39.333542Z info FLAG: --log_caller="" 2026-11-03T16:32:39.333544Z info FLAG: --log_output_level="default:info" 2026-11-03T16:32:39.333547Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T16:32:39.333551Z info FLAG: --log_target="[stdout]" 2026-11-03T16:32:39.333554Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T16:32:39.333556Z info FLAG: --outlierLogPath="" 2026-11-03T16:32:39.333557Z info FLAG: --profiling="true" 2026-11-03T16:32:39.333559Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T16:32:39.333562Z info FLAG: --proxyLogLevel="warning" 2026-11-03T16:32:39.333565Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T16:32:39.333567Z info FLAG: --stsPort="0" 2026-11-03T16:32:39.333569Z info FLAG: --templateFile="" 2026-11-03T16:32:39.333571Z info FLAG: --tokenManagerPlugin="" 2026-11-03T16:32:39.333573Z info FLAG: --vklog="0" 2026-11-03T16:32:39.333576Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T16:32:39.333788Z info Proxy role ips=[10.133.0.47] type=router id=maas-default-gateway-openshift-default-c9ffffc7f-zvhjn.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T16:32:39.333834Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T16:32:39.335069Z info cpu limit detected as 2, setting concurrency 2026-11-03T16:32:39.335295Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T16:32:39.335307Z info JWT policy is third-party-jwt 2026-11-03T16:32:39.335310Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T16:32:39.336764Z info Starting default Istio SDS Server 2026-11-03T16:32:39.336795Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T16:32:39.336819Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T16:32:39.337122Z info Opening status port 15020 2026-11-03T16:32:39.338748Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T16:32:39.340146Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T16:32:39.340975Z info Starting proxy agent 2026-11-03T16:32:39.341008Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T16:32:39.348825Z info sds Starting SDS grpc server 2026-11-03T16:32:39.348852Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T16:32:39.404772Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T16:32:39.516710Z info cache generated new workload certificate resourceName=default latency=177.545017ms ttl=23h59m59.483294502s 2026-11-03T16:32:39.516747Z info cache Root cert has changed, start rotating root cert 2026-11-03T16:32:39.516784Z info cache returned workload trust anchor from cache ttl=23h59m59.483216721s 2026-11-03T16:32:39.532721Z info ads ADS: new connection for node:1 2026-11-03T16:32:39.532781Z info cache returned workload certificate from cache ttl=23h59m59.46722016s 2026-11-03T16:32:39.533463Z info ads ADS: new connection for node:2 2026-11-03T16:32:39.533800Z info cache returned workload trust anchor from cache ttl=23h59m59.466202463s 2026-11-03T16:32:39.581161Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:32:45.925565Z info Readiness succeeded in 6.594980915s 2026-11-03T16:32:45.925858Z info Envoy proxy is ready 2026-11-03T16:32:54.007113Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:32:58.076969Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:33:00.236301Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:33:02.822717Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:33:04.974707Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:33:07.961844Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:33:13.038906Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:33:16.750058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T16:33:43.577Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 62 - "44.212.242.249,10.132.0.10" "curl/7.76.1" "f27573f4-92ac-4ef1-b283-2de3a38ff622" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.47:443 10.132.0.10:37696 - v1-model-routing 2026-11-03T16:37:12.268187Z info ads ADS: new connection for node:3 2026-11-03T16:37:12.284443Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T16:37:12.284539Z info cache read certificate from file resource=file-root:system 2026-11-03T16:37:13.696943Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:37:21.502554Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:37:28.642188Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:37:31.843135Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T16:37:31.843173Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T16:37:31.843188Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T16:37:31.843193Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T16:37:31.435Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 74 316 407 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "ec8c9ba0-d9ed-496c-b183-0dc4f5f932cb" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:41750 10.133.0.47:443 10.132.0.10:57950 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T16:37:39.845715Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:37:54.100966Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:37:54.101007Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:37:54.101022Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T16:37:54.101028Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:37:54.101281Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T16:37:53.982Z] "POST /anything/e2e-rt-sw-3c8550-client HTTP/1.1" 200 - via_upstream - "-" 124 1755 118 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "63e421f7-087a-4497-8640-e131f360c2ae" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.47:48124 10.133.0.47:443 10.132.0.10:40120 - llm.external-model-e2e-rt-sw-3c8550-b4ff46527fe3.0 2026-11-03T16:37:55.211489Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:38:08.196193Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:38:30.597271Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:38:30.631163Z info ads ADS: "" 3 terminated with stream closed 2026-11-03T16:38:36.514672Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:41:02.853144Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:43:02.792335Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:43:07.158549Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T16:43:07.158582Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T16:43:07.158597Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T16:43:07.158603Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T16:43:07.616919Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:43:07.616964Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:43:07.616985Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T16:43:07.616991Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:43:07.746381Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:43:07.746418Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:43:07.746463Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:43:07.746468Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T16:43:07.024Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 134 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "3bb9127a-87c0-4ca7-8690-12898310dc2b" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:57808 10.133.0.47:443 10.132.0.10:35376 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T16:43:07.384Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 232 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "90853287-536a-4232-8ac3-4bac78c41f19" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:57822 10.133.0.47:443 10.132.0.10:35392 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T16:43:07.664Z] "DELETE /v1/api-keys/a1c6737f-fef5-4963-babb-e7e49af66da0 HTTP/1.1" 200 - via_upstream - "-" 0 321 82 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "671f6497-646d-42d8-85c9-8d6ff19829a2" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:57822 10.133.0.47:443 10.132.0.10:35416 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T16:43:29.100170Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T16:43:41.399652Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:43:41.399689Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:43:41.399703Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:43:41.399708Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T16:43:41.822482Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-03T16:43:41.321Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 78 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "9bdc3292-d00e-40fb-96d6-9c15a467ad3d" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:57822 10.133.0.47:443 10.132.0.10:55714 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T16:43:41.760Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 77 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "cf6c7e6b-61ff-4378-b9a1-4a044636974f" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:35962 10.133.0.47:443 10.132.0.10:55718 - v1-completions-path 2026-11-03T16:54:02.823891Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: gRPC status code is not OK thread=24 [2026-11-03T16:54:00.808Z] "POST /maas-api/v1/api-keys HTTP/1.1" 500 - - "-" 72 23 2016 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "2649d559-7773-4d87-8be2-87e5ddac513b" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.47:443 10.132.0.10:57698 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T16:54:08.721569Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:54:08.721621Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:54:08.721650Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T16:54:08.721658Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T16:54:07.868Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 314 853 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "fd1ffa33-24f0-4364-8228-b28ae45e00ba" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:47890 10.133.0.47:443 10.132.0.10:51004 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T16:59:14.430Z] "GET /v1/tenants HTTP/1.1" 404 NR route_not_found - "-" 0 0 12 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "84ae3fb3-026b-4959-af98-f1ebebda39cd" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "-" - - 10.133.0.47:443 10.132.0.10:43440 - - 2026-11-03T17:02:45.677891Z info ads ADS: new connection for node:4 2026-11-03T17:02:45.683873Z info cache read certificate from file resource=file-root:system 2026-11-03T17:03:35.588804Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-03T17:09:32.472677Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:09:55.140510Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:10:13.693298Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:10:17.361233Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:10:22.259620Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:10:26.105669Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:10:31.543254Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:10:35.413408Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:11:06.749410Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:12:34.553281Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:12:39.020931Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:12:55.634060Z info ads ADS: new connection for node:5 2026-11-03T17:12:55.634443Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T17:12:55.634541Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T17:13:51.006235Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:14:23.700173Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:14:36.256818Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:14:36.256851Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:14:36.256861Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:14:36.256866Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T17:14:36.257019Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T17:14:36.132Z] "POST /anything/e2e-dft-611580-8ae3dc HTTP/1.1" 200 - via_upstream - "-" 109 1763 124 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "79f1312a-97d9-41e5-88d5-1b125cd82305" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.47:51548 10.133.0.47:443 10.132.0.10:36222 - llm.external-model-e2e-dft-611580-0732e92824a7.0 2026-11-03T17:14:48.897842Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:15:24.065587Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:16:53.914071Z info ads ADS: "" 5 terminated with stream closed 2026-11-03T17:16:56.653801Z info ads ADS: "" 4 terminated 2026-11-03T17:21:27.646011Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-11-03T17:27:09.227855Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:27:44.717520Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:28:14.611371Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:29:53.888551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:33:29.563232Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-03T17:39:00.954586Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:39:24.595250Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:39:58.089282Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T17:40:01.961Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 140 405 87 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "8657c527-1375-4926-95ed-3d4561eb51a1" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.134.0.42:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:54948 10.133.0.47:443 10.132.0.10:48458 - v1-chat-completions-path 2026-11-03T17:40:14.016532Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:40:16.569733Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:40:19.412444Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T17:40:19.433638Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-03T17:40:19.353Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 80 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "95661911-41e3-4341-912f-3141fe49eafa" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.42:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:37686 10.133.0.47:443 10.132.0.10:37122 - v1-catch-all-path 2026-11-03T17:40:22.995003Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:40:30.284072Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:40:39.971723Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:40:54.738588Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:41:05.770842Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:41:07.693612Z info ads ADS: new connection for node:6 2026-11-03T17:41:07.693906Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T17:41:07.693981Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T17:41:09.738997Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:41:50.167493Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T17:41:55.058Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 329 86 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "4a6c7ebe-5c14-4c66-af24-44ea386689d9" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.133.0.210:8000" outbound|54321||e2e-prio-pool-39e7fadb-inference-pool-ip-afbec136.llm.svc.cluster.local 10.133.0.47:52136 10.133.0.47:443 10.132.0.10:47338 - v1-model-routing 2026-11-03T17:42:09.206759Z info ads ADS: "" 6 terminated 2026-11-03T17:42:09.263727Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T17:42:11.533Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 69 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "963838d8-ed9b-40eb-b172-3f4d019a669f" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:60970 10.133.0.47:443 10.132.0.10:60584 - v1-completions-path 2026-11-03T17:42:12.685468Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:42:19.774473Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:44:34.326066Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:44:52.378357Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:45:08.321415Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:45:19.289258Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:45:26.761706Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:45:38.600316Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:45:42.225832Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:45:45.052028Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:45:45.052075Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:45:45.052098Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:45:45.052106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T17:45:44.966Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 93 335 85 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "b6c30e0b-5189-42a4-8027-ea7890fac861" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:53134 10.133.0.47:443 10.132.0.10:34618 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T17:45:57.829408Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:46:44.301970Z info ads ADS: new connection for node:7 2026-11-03T17:46:44.307517Z info cache read certificate from file resource=file-root:system 2026-11-03T17:46:45.467540Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:46:47.048145Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:46:52.068127Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"1187","StreamId":"338735162687363249"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-03T17:46:52.004Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 63 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "dd69074f-a768-447a-9068-a83c10f01ac5" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.47:443 10.132.0.10:44368 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T17:46:58.368179Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:46:58.368215Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:46:58.368227Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:46:58.368232Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T17:46:58.368387Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T17:46:58.256Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1707 111 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "b3efb9de-372b-48d3-81e7-d0977bf76842" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.47:37022 10.133.0.47:443 10.132.0.10:58114 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T17:47:15.559362Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:47:15.559398Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:47:15.559409Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:47:15.559413Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T17:47:15.559613Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T17:47:15.460Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1713 99 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "f862aab7-2d93-4035-b4c4-07de08045312" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.47:37022 10.133.0.47:443 10.132.0.10:41922 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T17:48:25.183307Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:48:25.302351Z info ads ADS: "" 7 terminated with stream closed 2026-11-03T17:48:26.793600Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:48:33.783406Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:48:36.457896Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:48:36.457932Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:48:36.457948Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:48:36.457953Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T17:48:36.330Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 87 329 127 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "9b81d2da-07e7-4da4-ba1b-fcc50d327773" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:59444 10.133.0.47:443 10.132.0.10:40054 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T17:48:37.125Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 73 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "8f9dc39d-1d1a-4c8f-8ec6-39486cd3fcb6" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:41286 10.133.0.47:443 10.132.0.10:40062 - v1-completions-path [2026-11-03T17:48:37.333Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "a43b7b7e-7be0-42ed-96f3-e0cb28e95a92" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:41286 10.133.0.47:443 10.132.0.10:40072 - v1-completions-path 2026-11-03T17:48:49.723700Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:48:54.316807Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:49:09.519414Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:49:11.940340Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:49:14.948213Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:49:25.043671Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T17:49:31.454Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 65 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "83f55ca2-f9b2-4fe7-8a5f-300e2cba3300" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:54984 10.133.0.47:443 10.132.0.10:33998 - v1-completions-path 2026-11-03T17:49:48.192687Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:49:58.542289Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:50:03.083781Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T17:50:07.328Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 61 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "002905cb-2b92-43ef-ab50-05bede06d029" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:54984 10.133.0.47:443 10.132.0.10:55630 - v1-completions-path [2026-11-03T17:50:07.491Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 62 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "2cf079d1-896c-4e8c-bc2f-928dc53789ff" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:54984 10.133.0.47:443 10.132.0.10:55634 - v1-completions-path 2026-11-03T17:50:19.936772Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:50:22.263329Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:50:29.322751Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:50:41.282557Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:50:46.384107Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:50:54.758887Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:50:59.909325Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:51:18.274767Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:51:21.299320Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:51:37.123922Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:51:40.076296Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T17:51:42.152Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 34 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "423234c8-9823-4dac-9c2e-9303a5345fcb" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.47:443 10.132.0.10:32840 - v1-completions-path 2026-11-03T17:51:49.054871Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:52:06.810190Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:52:17.115752Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:52:34.660896Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:52:38.013502Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:54:57.161136Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:55:00.570766Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:56:05.073353Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T17:56:05.073394Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T17:56:05.073417Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T17:56:05.073442Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T17:56:04.993Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 70 313 79 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "de7aeae0-c6b9-4b70-93cc-d35df3662867" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:51228 10.133.0.47:443 10.132.0.10:49080 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T17:56:25.510962Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:56:40.971218Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:56:43.375515Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:56:43.447793Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T17:56:43.447925Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T17:56:43.447981Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T17:56:43.448013Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T17:56:43.346Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 65 307 101 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "b6f8e9f8-363e-4dd7-a2f0-786d23fde69c" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.35:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:51228 10.133.0.47:443 10.132.0.10:55390 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T17:56:43.469Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 292 80 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "42211dd3-e753-417e-86b8-ea22a817babf" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:60970 10.133.0.47:443 10.132.0.10:55394 - v1-completions-path 2026-11-03T17:56:50.695735Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:57:07.532442Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:57:10.603609Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:57:14.011991Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:57:14.012026Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:57:14.012042Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T17:57:14.012047Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T17:57:13.482Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 64 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "6af05487-f971-426d-bc49-9807caf8e306" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:41286 10.133.0.47:443 10.132.0.10:55032 - v1-completions-path [2026-11-03T17:57:13.936Z] "DELETE /v1/api-keys/9f2af858-3a12-48ad-951a-38d4a6c22383 HTTP/1.1" 200 - via_upstream - "-" 0 369 75 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "3b5a03e3-4870-4529-961a-5996a7a6fe72" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.133.0.219:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.47:53882 10.133.0.47:443 10.132.0.10:55040 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T17:57:26.017124Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:57:37.166285Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:57:41.813609Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T17:57:45.618Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 68 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "f8de7413-8957-46c5-bd85-855f77614b94" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:41286 10.133.0.47:443 10.132.0.10:35422 - v1-completions-path 2026-11-03T17:57:58.838884Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:58:03.564402Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:58:08.796822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T17:58:16.367Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 75 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "459fbf25-1ce3-4067-8738-b91364f7f3a3" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:41286 10.133.0.47:443 10.132.0.10:41204 - v1-completions-path [2026-11-03T17:58:18.314Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 60 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "207ea685-e6c3-493c-96ad-fd902a7e241b" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:41286 10.133.0.47:443 10.132.0.10:41222 - v1-completions-path 2026-11-03T17:58:31.689592Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:58:41.963583Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:58:46.492893Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:58:49.474445Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:58:52.471317Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T17:58:56.356Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "c5591588-4c0a-44d9-ae25-fa91538b0123" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.43:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:54984 10.133.0.47:443 10.132.0.10:51136 - v1-completions-path 2026-11-03T17:59:09.830887Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:59:20.331370Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:59:22.536631Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:59:30.890372Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:59:34.551365Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:59:52.626419Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T17:59:55.003919Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T18:00:13.292668Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T18:00:23.663086Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T18:00:33.975519Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T18:00:44.371230Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T18:00:54.750243Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T18:01:05.144730Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T18:01:15.607950Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T18:01:25.871312Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T18:01:36.250878Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T18:01:46.565392Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T18:01:55.601Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 64 - "44.212.242.249,10.132.0.10" "python-requests/2.34.2" "eba358c8-8d55-4bd9-84c8-30d40c38f251" "maas.apps.38d07d575d4ce9451258.prod.konflux-ocp-ci.dev" "10.132.0.44:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.47:60970 10.133.0.47:443 10.132.0.10:52080 - v1-completions-path 2026-11-03T18:02:06.682539Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=4