2026-10-30T07:42:35.744507Z info FLAG: --concurrency="0" 2026-10-30T07:42:35.744527Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T07:42:35.744532Z info FLAG: --help="false" 2026-10-30T07:42:35.744535Z info FLAG: --log_as_json="false" 2026-10-30T07:42:35.744538Z info FLAG: --log_caller="" 2026-10-30T07:42:35.744540Z info FLAG: --log_output_level="default:info" 2026-10-30T07:42:35.744543Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T07:42:35.744550Z info FLAG: --log_target="[stdout]" 2026-10-30T07:42:35.744553Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T07:42:35.744556Z info FLAG: --outlierLogPath="" 2026-10-30T07:42:35.744559Z info FLAG: --profiling="true" 2026-10-30T07:42:35.744562Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T07:42:35.744564Z info FLAG: --proxyLogLevel="warning" 2026-10-30T07:42:35.744567Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T07:42:35.744570Z info FLAG: --stsPort="0" 2026-10-30T07:42:35.744573Z info FLAG: --templateFile="" 2026-10-30T07:42:35.744576Z info FLAG: --tokenManagerPlugin="" 2026-10-30T07:42:35.744580Z info FLAG: --vklog="0" 2026-10-30T07:42:35.744584Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T07:42:35.744778Z info Proxy role ips=[10.134.0.54] type=router id=maas-default-gateway-openshift-default-67bf986f99-lscxm.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T07:42:35.744848Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T07:42:35.746184Z info cpu limit detected as 2, setting concurrency 2026-10-30T07:42:35.746447Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T07:42:35.746460Z info JWT policy is third-party-jwt 2026-10-30T07:42:35.746464Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T07:42:35.747916Z info Starting default Istio SDS Server 2026-10-30T07:42:35.747957Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T07:42:35.747979Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T07:42:35.748786Z info Opening status port 15020 2026-10-30T07:42:35.749288Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T07:42:35.750113Z info sds Starting SDS grpc server 2026-10-30T07:42:35.750132Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T07:42:35.751028Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T07:42:35.751908Z info Starting proxy agent 2026-10-30T07:42:35.751961Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T07:42:35.827755Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T07:42:35.874561Z info cache generated new workload certificate resourceName=default latency=124.30821ms ttl=23h59m59.125442699s 2026-10-30T07:42:35.874592Z info cache Root cert has changed, start rotating root cert 2026-10-30T07:42:35.874633Z info cache returned workload trust anchor from cache ttl=23h59m59.125367788s 2026-10-30T07:42:35.960119Z info ads ADS: new connection for node:1 2026-10-30T07:42:35.960217Z info cache returned workload certificate from cache ttl=23h59m59.039786052s 2026-10-30T07:42:35.960237Z info ads ADS: new connection for node:2 2026-10-30T07:42:35.960332Z info cache returned workload trust anchor from cache ttl=23h59m59.039669308s 2026-10-30T07:42:36.049142Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:42:43.827016Z info Readiness succeeded in 8.084936985s 2026-10-30T07:42:43.827439Z info Envoy proxy is ready 2026-10-30T07:43:05.338505Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:22.171513Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:25.107643Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:32.720686Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-10-30T07:43:32.632Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 88 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "4dde219e-c61d-4ee7-bfc9-23f5c1534abe" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.133.0.38:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.54:55942 10.134.0.54:443 10.132.0.11:48936 - v1-catch-all-path 2026-10-30T07:43:32.963353Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-10-30T07:43:32.906Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 57 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "6582910f-3ff8-4845-992a-dcc44554e779" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.133.0.38:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.54:55942 10.134.0.54:443 10.132.0.11:48944 - v1-catch-all-path 2026-10-30T07:43:35.480174Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:53.586620Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:43:55.896128Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:10.622816Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:21.657090Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:44:24.147166Z info ads ADS: new connection for node:3 2026-10-30T07:44:24.147366Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T07:44:24.147398Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T07:44:28.049520Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:06.637710Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:45:24.800Z] "POST /v1/chat/completions HTTP/1.1" 403 - - "-" 131 0 45 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "3dcd57f6-447e-483c-b169-bac5062b2686" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "-" outbound|54321||e2e-prio-pool-74f9b463-inference-pool-ip-f42fb901.llm.svc.cluster.local - 10.134.0.54:443 10.132.0.11:49124 - v1-model-routing 2026-10-30T07:45:26.170187Z info ads ADS: "" 3 terminated with stream closed 2026-10-30T07:45:26.223595Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:28.570212Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:31.422495Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:45:36.854606Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:07.424750Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:47:07.424792Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:47:07.424812Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:47:07.424820Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T07:47:07.333Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 33 326 91 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "69fe5a58-88bc-4600-a016-08d7f2600efa" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.132.0.53:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.54:57964 10.134.0.54:443 10.132.0.11:47296 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T07:47:13.122756Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:29.747079Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:45.435700Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:47:57.445206Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:48:01.601954Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:48:12.759862Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:48:17.179893Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:48:35.709865Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:49:26.406892Z info ads ADS: new connection for node:4 2026-10-30T07:49:26.413271Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T07:49:26.413327Z info cache read certificate from file resource=file-root:system 2026-10-30T07:49:27.597435Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:49:30.088172Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:49:37.148732Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"130","StreamId":"6597658192344052555"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-10-30T07:49:37.088Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 60 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "64c33549-4275-488f-b239-2c309f4784e3" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.134.0.54:443 10.132.0.11:50970 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-10-30T07:49:41.375862Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:49:41.375906Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:49:41.375917Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:49:41.375921Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T07:49:41.376104Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T07:49:41.264Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1730 111 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "02554f44-ce7f-4935-9b18-6c60f15488df" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.54:48348 10.134.0.54:443 10.132.0.11:50996 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:49:52.452721Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:49:52.452748Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:49:52.452757Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T07:49:52.452761Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:49:52.452900Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T07:49:52.353Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1731 99 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "b3c5830e-74a7-4ded-8f81-206afd9fdc47" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.54:48348 10.134.0.54:443 10.132.0.11:59452 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:49:58.789025Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-10-30T07:49:58.886660Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:49:58.886689Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:49:58.886699Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T07:49:58.886703Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T07:49:58.886843Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T07:49:58.783Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1731 103 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "143bbe5d-e2eb-41c8-976a-80bfb6ff1087" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.54:47632 10.134.0.54:443 10.132.0.11:51754 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:50:01.010322Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:50:01.010356Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:50:01.010366Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T07:50:01.010371Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:50:01.010519Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T07:50:00.915Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1731 94 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "16bd93cc-a9fd-445c-9529-2d9bd3657804" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.54:48348 10.134.0.54:443 10.132.0.11:51766 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-10-30T07:50:55.186189Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:50:55.285569Z info ads ADS: "" 4 terminated 2026-10-30T07:50:55.774287Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:50:58.781973Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T07:50:58.781998Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T07:50:58.782003Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T07:50:58.782009Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 2026-10-30T07:50:58.928130Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:50:58.717Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 64 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "7b5a2fa7-b143-4d17-924a-dee67911de65" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.54:443 10.132.0.11:42736 - v1-completions-path 2026-10-30T07:51:04.152119Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:23.999744Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:28.356069Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:51:44.240398Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:00.111822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:05.198401Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-30T07:52:05.270278Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:52:05.270314Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:52:05.270329Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T07:52:05.270334Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T07:52:05.190Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 80 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "d1920056-27fb-4c3b-9b2c-e7769dcf0800" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.132.0.53:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.54:58166 10.134.0.54:443 10.132.0.11:42802 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T07:52:06.361Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 58 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "30a84b1c-fea7-4681-bf73-55afe1e1dfdf" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.54:443 10.132.0.11:42808 - v1-completions-path [2026-10-30T07:52:06.689Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 69 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "a27b3e17-9c5a-45a6-a7a0-f4ef9cd78e38" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.133.0.39:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.54:54722 10.134.0.54:443 10.132.0.11:42812 - v1-completions-path 2026-10-30T07:52:23.651595Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:34.013990Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:38.725840Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T07:52:43.200Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 62 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "67546d33-3ba9-419b-b1af-c593bf46fefa" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.133.0.39:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.54:54722 10.134.0.54:443 10.132.0.11:34956 - v1-completions-path 2026-10-30T07:52:55.703928Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:52:58.000716Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:17.078660Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:21.669087Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:37.286847Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:39.629857Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:53:57.508997Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:09.179447Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:12.977605Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:41.250854Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:54:59.392999Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:55:09.803513Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:57:11.541105Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:57:15.243884Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:58:46.475290Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:59:02.351454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:59:04.740591Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:59:11.752345Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T07:59:11.752371Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T07:59:11.752374Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T07:59:11.752378Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T07:59:11.714Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 37 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "94d5008a-19bc-4e4a-8cc8-bc861c459769" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.54:443 10.132.0.11:39752 - v1-completions-path 2026-10-30T07:59:23.696493Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:59:36.072031Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:59:39.462363Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T07:59:55.510256Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:00:06.107654Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:00:11.176701Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:00:28.052155Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:00:32.770994Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:00:35.590816Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T08:00:44.015Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 63 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "c326109e-2f09-4aee-8fe2-0ab7ffdac855" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.133.0.39:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.54:54722 10.134.0.54:443 10.132.0.11:41390 - v1-completions-path [2026-10-30T08:00:44.390Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 61 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "a73d526b-c95f-48b4-9d0b-bc9dfcbfe761" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.133.0.39:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.54:59250 10.134.0.54:443 10.132.0.11:41398 - v1-completions-path [2026-10-30T08:00:44.574Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 59 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "f14e88b6-f084-4bba-9e3c-d2e8535a7bfa" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.133.0.39:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.54:59250 10.134.0.54:443 10.132.0.11:41402 - v1-completions-path 2026-10-30T08:00:58.878153Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:01:09.131951Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:01:13.747482Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:01:17.159454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:01:22.088150Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T08:01:27.732Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 61 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "3109f69d-edca-4303-8620-0792d47303b6" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.133.0.39:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.54:54722 10.134.0.54:443 10.132.0.11:54990 - v1-completions-path [2026-10-30T08:01:27.915Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 56 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "2fe2ac1a-775b-407d-8602-3d90384cf0fe" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.133.0.39:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.54:54722 10.134.0.54:443 10.132.0.11:55004 - v1-completions-path 2026-10-30T08:01:40.373867Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:01:50.846268Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:01:53.201586Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:02:03.376687Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:02:08.110801Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:02:24.823305Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:02:44.681917Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:02:55.121647Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:03:05.602048Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:03:15.962116Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:03:26.282705Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:03:36.645696Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:03:47.001666Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:03:57.215091Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:04:07.618423Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-10-30T08:04:18.143866Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-10-30T08:04:27.214Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 287 80 - "34.228.250.194,10.132.0.11" "python-requests/2.34.2" "71ab4d53-d9e5-48d4-b478-18bf59e78979" "maas.apps.59c8f9f7001e17a9d6fe.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.54:59196 10.134.0.54:443 10.132.0.11:34398 - v1-completions-path