2026-10-30T06:39:22.793280Z info FLAG: --concurrency="0" 2026-10-30T06:39:22.793306Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T06:39:22.793310Z info FLAG: --help="false" 2026-10-30T06:39:22.793313Z info FLAG: --log_as_json="false" 2026-10-30T06:39:22.793315Z info FLAG: --log_caller="" 2026-10-30T06:39:22.793317Z info FLAG: --log_output_level="default:info" 2026-10-30T06:39:22.793319Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T06:39:22.793323Z info FLAG: --log_target="[stdout]" 2026-10-30T06:39:22.793326Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T06:39:22.793328Z info FLAG: --outlierLogPath="" 2026-10-30T06:39:22.793330Z info FLAG: --profiling="true" 2026-10-30T06:39:22.793332Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T06:39:22.793334Z info FLAG: --proxyLogLevel="warning" 2026-10-30T06:39:22.793337Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T06:39:22.793339Z info FLAG: --stsPort="0" 2026-10-30T06:39:22.793341Z info FLAG: --templateFile="" 2026-10-30T06:39:22.793343Z info FLAG: --tokenManagerPlugin="" 2026-10-30T06:39:22.793346Z info FLAG: --vklog="0" 2026-10-30T06:39:22.793348Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T06:39:22.793491Z info Proxy role ips=[10.133.0.90] type=router id=maas-default-gateway-openshift-default-85544d4cd7-mj669.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T06:39:22.793534Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T06:39:22.794582Z info cpu limit detected as 2, setting concurrency 2026-10-30T06:39:22.794778Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T06:39:22.794789Z info JWT policy is third-party-jwt 2026-10-30T06:39:22.794792Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T06:39:22.796440Z info Starting default Istio SDS Server 2026-10-30T06:39:22.796463Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T06:39:22.796485Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T06:39:22.796571Z info Opening status port 15020 2026-10-30T06:39:22.797273Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T06:39:22.799229Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T06:39:22.801084Z info sds Starting SDS grpc server 2026-10-30T06:39:22.801102Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T06:39:22.801272Z info Starting proxy agent 2026-10-30T06:39:22.801299Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T06:39:22.884769Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T06:39:22.977067Z info cache generated new workload certificate resourceName=default latency=179.474258ms ttl=23h59m59.022939273s 2026-10-30T06:39:22.977114Z info cache Root cert has changed, start rotating root cert 2026-10-30T06:39:22.977156Z info cache returned workload trust anchor from cache ttl=23h59m59.022845186s 2026-10-30T06:39:23.058158Z info ads ADS: new connection for node:1 2026-10-30T06:39:23.058267Z info cache returned workload certificate from cache ttl=23h59m58.941736506s 2026-10-30T06:39:23.058482Z info ads ADS: new connection for node:2 2026-10-30T06:39:23.058557Z info cache returned workload trust anchor from cache ttl=23h59m58.941444073s 2026-10-30T06:39:23.218214Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:39:31.400742Z info Readiness succeeded in 8.609907726s 2026-10-30T06:39:31.401056Z info Envoy proxy is ready 2026-10-30T06:40:32.410797Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:40:53.003Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 67 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "2b1fe51a-9015-4065-a73d-1f315c686e25" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.90:443 10.132.0.21:41336 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:40:56.060377Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:40:56.060433Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T06:40:56.060456Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T06:40:56.060463Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T06:40:55.953Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 107 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "42ef1f6b-ae63-4618-904f-2f08be7d039c" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.90:50254 10.133.0.90:443 10.132.0.21:41352 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T06:41:04.608Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 97 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "da6921af-e865-4cbc-8af3-f40c3ed7c6bf" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:51182 10.133.0.90:443 10.132.0.21:46342 - v1-completions-path 2026-10-30T06:41:10.274464Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T06:41:10.274499Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T06:41:10.274513Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-30T06:41:10.274517Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-10-30T06:41:10.200Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 73 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "1f93dc55-dfd9-4ff6-9459-b496651af7b7" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.90:41970 10.133.0.90:443 10.132.0.21:54586 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T06:41:14.879Z] "POST /maas-api/internal/v1/subscriptions/select HTTP/1.1" 404 NR route_not_found - "-" 56 0 9 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "b6157889-d287-4790-9aba-23c87493256f" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" - - 10.133.0.90:443 10.132.0.21:54600 - - 2026-10-30T06:41:17.759658Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T06:41:17.759701Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T06:41:17.759722Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T06:41:17.759728Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-10-30T06:41:17.687Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 72 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "ca2fbdad-eb97-4057-b3c8-b6f55a73b434" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.90:41970 10.133.0.90:443 10.132.0.21:54616 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:41:30.602111Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:45:03.935965Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T06:45:27.107526Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T06:48:15.842Z] "POST /maas-api/v1/tokens HTTP/1.1" 401 - - "-" 20 88 24 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "54946a71-c7ff-4c01-8b7d-e04bb1f204ad" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.133.0.90:443 10.132.0.21:36816 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:51:30.727789Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T06:51:30.727831Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T06:51:30.727853Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T06:51:30.727860Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-10-30T06:51:30.556Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 171 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "f75e1190-8cc6-4f10-9d2c-e3ab6edcb93e" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.90:48736 10.133.0.90:443 10.132.0.21:35340 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T06:59:12.704792Z info ads ADS: new connection for node:3 2026-10-30T06:59:12.711152Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T06:59:12.711211Z info cache read certificate from file resource=file-root:system 2026-10-30T07:03:58.485734Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:04:17.413322Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:04:35.953831Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:04:51.258461Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:04:54.751374Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:04:59.595150Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:05:07.097663Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:05:11.530847Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:05:14.373081Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:05:16.757617Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:05:20.982695Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:05:49.416663Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:08:54.681529Z info ads ADS: new connection for node:4 2026-10-30T07:08:54.681839Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T07:08:54.681986Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T07:10:38.100592Z info ads ADS: "" 4 terminated 2026-10-30T07:11:01.824042Z info ads ADS: "" 3 terminated with stream closed 2026-10-30T07:11:40.690320Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T07:21:11.903926Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:21:52.683272Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:23:32.827417Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:23:53.797842Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:25:37.096741Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:27:11.581305Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-10-30T07:27:23.509515Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-10-30T07:34:06.454071Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:34:06.454106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:34:06.454121Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:34:06.454126Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T07:34:06.361Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 30 315 93 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "2b56d98c-db8f-4d73-aa10-54d484125e29" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.90:36390 10.133.0.90:443 10.132.0.21:56962 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T07:34:09.785737Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:34:33.795153Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:35:08.361359Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:35:26.521093Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:35:28.748510Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:35:39.659575Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:35:45.085766Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-10-30T07:35:45.010Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 75 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "d487cc94-d1cb-4d0d-a82c-328565609bb4" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:35542 10.133.0.90:443 10.132.0.21:43546 - v1-catch-all-path 2026-10-30T07:35:57.700232Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:36:00.015735Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:36:02.867650Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 2026-10-30T07:36:02.868798Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=26 [2026-10-30T07:36:02.805Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 62 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "471bc66e-5065-4447-9c19-1b8ac6374bb6" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:42548 10.133.0.90:443 10.132.0.21:37538 - v1-catch-all-path 2026-10-30T07:36:14.808598Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:36:25.842957Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:36:27.799335Z info ads ADS: new connection for node:5 2026-10-30T07:36:27.799687Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T07:36:27.799792Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T07:36:29.916630Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:37:10.388717Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:37:29.377551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:37:29.464984Z info ads ADS: "" 5 terminated 2026-10-30T07:37:33.031305Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:37:40.621915Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:39:16.022552Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:39:32.492466Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:39:47.815574Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:40:01.887045Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:40:06.747846Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:40:07.367400Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:40:07.367449Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:40:07.367471Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-30T07:40:07.367479Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-10-30T07:40:07.267Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 99 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "fd640360-adb8-42d5-bb55-7e917f94c40d" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.90:35882 10.133.0.90:443 10.132.0.21:36444 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T07:40:19.300800Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:40:22.907730Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:40:43.057294Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:41:30.281598Z info ads ADS: new connection for node:6 2026-10-30T07:41:30.292916Z info cache read certificate from file resource=file-root:system 2026-10-30T07:41:31.417241Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:41:32.960953Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:41:39.694379Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"996","StreamId":"2609456168130935794"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=26 [2026-10-30T07:41:39.629Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 65 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "0e5762e8-6d83-4f4e-97bb-41b32f19bf8e" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.90:443 10.132.0.21:55698 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T07:41:52.573698Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:41:52.573735Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:41:52.573746Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:41:52.573751Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-30T07:41:52.573955Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-10-30T07:41:52.467Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1728 106 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "5b3aa5a5-3d86-4298-9cb1-183ad1b67b16" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.90:51786 10.133.0.90:443 10.132.0.21:38622 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:42:05.357132Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:42:05.357163Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:42:05.357174Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-30T07:42:05.357178Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:42:05.357346Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-10-30T07:42:05.254Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 102 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "94054b5d-e7b6-41d8-b463-55ced92f654a" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.90:51786 10.133.0.90:443 10.132.0.21:35882 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:42:11.785222Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:42:11.785258Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:42:11.785279Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-30T07:42:11.785284Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:42:11.785462Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-10-30T07:42:11.688Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1733 96 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "9eef024c-fbd5-42cf-95e3-71ae377bd49d" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.90:51786 10.133.0.90:443 10.132.0.21:41770 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:42:20.258695Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-30T07:42:20.305676Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:42:20.305732Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:42:20.305749Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:42:20.305754Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T07:42:20.306014Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T07:42:20.197Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1727 108 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "9fc24645-1cd8-4e1f-bbc1-163db99dd692" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.90:51312 10.133.0.90:443 10.132.0.21:55200 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:42:39.567072Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:42:39.567106Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:42:39.567119Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:42:39.567124Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-10-30T07:42:39.567316Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-10-30T07:42:39.465Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1729 102 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "156f398c-33c5-4435-ad3d-dc0bcb27f2b6" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.90:51786 10.133.0.90:443 10.132.0.21:54056 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:42:59.299193Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:42:59.449665Z info ads ADS: "" 6 terminated 2026-10-30T07:43:00.876997Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:07.765666Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:24.971334Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:29.471087Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:45.212164Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:01.047422Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:44:05.991Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 77 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "e2cace48-2b40-4a2d-a2e0-07ae9a368aa3" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:53172 10.133.0.90:443 10.132.0.21:56808 - v1-completions-path 2026-10-30T07:44:23.576608Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:33.849908Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:38.718268Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:44:45.849Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 57 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "80e9608b-1272-4b81-afb6-d388cf1234de" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:53172 10.133.0.90:443 10.132.0.21:45626 - v1-completions-path [2026-10-30T07:44:46.011Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 60 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "948b8bae-643b-4e61-bec9-af81e84c72d2" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:33822 10.133.0.90:443 10.132.0.21:45636 - v1-completions-path 2026-10-30T07:44:58.741614Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:01.017018Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:45:08.325Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 74 291 76 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "d48d7222-a836-495d-b798-47fce435ee46" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:45264 10.133.0.90:443 10.132.0.21:44072 - v1-completions-path 2026-10-30T07:45:20.573175Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:45:20.124Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 68 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "08f6d590-9329-4cd5-b244-c297ebadb767" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:43082 10.133.0.90:443 10.132.0.21:50008 - v1-completions-path 2026-10-30T07:45:22.884779Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:41.536667Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:43.847332Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:46:01.730091Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:46:12.393388Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:46:15.285009Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:46:21.826Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 63 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "c972e3da-a62d-418e-91a8-181f00c75a5c" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:33526 10.133.0.90:443 10.132.0.21:39646 - v1-completions-path 2026-10-30T07:46:34.593935Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:46:37.405078Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:46:42.970Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 33 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "34c6ff1e-ef79-42df-bfe4-a380c92bb31b" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.90:443 10.132.0.21:34804 - v1-completions-path 2026-10-30T07:46:43.097284Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:46:43.097319Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:46:43.097332Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:46:43.097338Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-10-30T07:46:43.026Z] "DELETE /v1/api-keys/1450615b-0bbb-40bf-87b1-d90700d853fd HTTP/1.1" 404 - via_upstream - "-" 0 29 71 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "f3a51e33-3783-4b25-b0fe-044bdc55fc39" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.90:55504 10.133.0.90:443 10.132.0.21:34836 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T07:46:50.002857Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:08.017221Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:18.354709Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:38.401835Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:41.781710Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:49:24.888997Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:49:28.379510Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:01.759456Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:17.263763Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:19.739916Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:26.564330Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-10-30T07:51:26.564355Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-10-30T07:51:26.564358Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=26 2026-10-30T07:51:26.564362Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=26 [2026-10-30T07:51:26.528Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 35 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "41aa1f44-3604-4d0f-82bb-5ed320cfb120" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.90:443 10.132.0.21:53284 - v1-completions-path 2026-10-30T07:51:38.615352Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:50.590402Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:53.554979Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:51:57.102Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 63 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "3836ac99-b9fb-4a35-b301-877b26fa35ef" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:53172 10.133.0.90:443 10.132.0.21:52488 - v1-completions-path 2026-10-30T07:52:09.440454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:19.992067Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:24.652620Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:52:29.504Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 59 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "d8961727-8f8a-4635-982e-a59b85aa4abc" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.90:443 10.132.0.21:38682 - v1-completions-path 2026-10-30T07:52:41.868389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:46.464847Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:49.550632Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:55.816705Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:52:55.816742Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:52:55.816758Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-10-30T07:52:55.816764Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-10-30T07:52:55.742Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 68 310 74 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "0cdcb14d-ec72-4893-b190-53e7ec9ae7ba" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.133.1.61:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.90:48802 10.133.0.90:443 10.132.0.21:57950 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T07:52:58.451Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 57 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "f0d3c1d0-7de2-4b48-abaa-8206b05ec803" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:33822 10.133.0.90:443 10.132.0.21:56640 - v1-completions-path [2026-10-30T07:52:58.811Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 56 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "2bce2956-d85d-40a0-8f72-52810eeeab9e" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:33822 10.133.0.90:443 10.132.0.21:56652 - v1-completions-path [2026-10-30T07:52:59.884Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 57 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "636046a4-1d7d-4844-b18b-333be56fae8d" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:33822 10.133.0.90:443 10.132.0.21:56668 - v1-completions-path 2026-10-30T07:53:12.521579Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:22.934222Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:27.477597Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:31.128245Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:36.183975Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:53:41.741Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 64 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "264af34f-27b5-4310-b848-eccb61a91b97" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:53172 10.133.0.90:443 10.132.0.21:53882 - v1-completions-path 2026-10-30T07:53:54.332067Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:04.707474Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:07.096905Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:18.456020Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:22.100436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:39.757210Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:42.192052Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:00.473218Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:10.927000Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:21.083091Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:31.471763Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:41.822002Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:52.338324Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:56:02.471354Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:56:12.941514Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:56:23.149174Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:56:33.496892Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:56:40.453Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 281 81 - "44.212.242.249,10.132.0.21" "python-requests/2.34.2" "4ca519fb-d751-42ed-b6b0-8921b30b8ed7" "maas.apps.66975bce54657ce23a19.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.90:33526 10.133.0.90:443 10.132.0.21:47538 - v1-completions-path