2026-11-01T06:10:50.618793Z info FLAG: --concurrency="0" 2026-11-01T06:10:50.618823Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-01T06:10:50.618827Z info FLAG: --help="false" 2026-11-01T06:10:50.618830Z info FLAG: --log_as_json="false" 2026-11-01T06:10:50.618832Z info FLAG: --log_caller="" 2026-11-01T06:10:50.618834Z info FLAG: --log_output_level="default:info" 2026-11-01T06:10:50.618836Z info FLAG: --log_stacktrace_level="default:none" 2026-11-01T06:10:50.618840Z info FLAG: --log_target="[stdout]" 2026-11-01T06:10:50.618844Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-01T06:10:50.618846Z info FLAG: --outlierLogPath="" 2026-11-01T06:10:50.618848Z info FLAG: --profiling="true" 2026-11-01T06:10:50.618850Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-01T06:10:50.618852Z info FLAG: --proxyLogLevel="warning" 2026-11-01T06:10:50.618855Z info FLAG: --serviceCluster="istio-proxy" 2026-11-01T06:10:50.618857Z info FLAG: --stsPort="0" 2026-11-01T06:10:50.618859Z info FLAG: --templateFile="" 2026-11-01T06:10:50.618861Z info FLAG: --tokenManagerPlugin="" 2026-11-01T06:10:50.618863Z info FLAG: --vklog="0" 2026-11-01T06:10:50.618866Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-01T06:10:50.619028Z info Proxy role ips=[10.133.1.56] type=router id=maas-default-gateway-openshift-default-58d898bd55-b4bkr.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-01T06:10:50.619081Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-01T06:10:50.620376Z info cpu limit detected as 2, setting concurrency 2026-11-01T06:10:50.620589Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-01T06:10:50.620601Z info JWT policy is third-party-jwt 2026-11-01T06:10:50.620603Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-01T06:10:50.622369Z info Starting default Istio SDS Server 2026-11-01T06:10:50.622403Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-01T06:10:50.622445Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-01T06:10:50.622867Z info Opening status port 15020 2026-11-01T06:10:50.623899Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-01T06:10:50.625773Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-01T06:10:50.626953Z info Starting proxy agent 2026-11-01T06:10:50.627013Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-01T06:10:50.634154Z info sds Starting SDS grpc server 2026-11-01T06:10:50.634177Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-01T06:10:50.703246Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-01T06:10:50.790948Z info cache generated new workload certificate resourceName=default latency=167.147163ms ttl=23h59m59.209058456s 2026-11-01T06:10:50.791001Z info cache Root cert has changed, start rotating root cert 2026-11-01T06:10:50.791042Z info cache returned workload trust anchor from cache ttl=23h59m59.208959015s 2026-11-01T06:10:50.874919Z info ads ADS: new connection for node:1 2026-11-01T06:10:50.875010Z info cache returned workload certificate from cache ttl=23h59m59.124993366s 2026-11-01T06:10:50.875523Z info ads ADS: new connection for node:2 2026-11-01T06:10:50.875620Z info cache returned workload trust anchor from cache ttl=23h59m59.12438194s 2026-11-01T06:10:50.959889Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:10:58.921075Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:11:00.097084Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-01T06:11:00.877633Z info Readiness succeeded in 10.261994733s 2026-11-01T06:11:00.877984Z info Envoy proxy is ready 2026-11-01T06:11:04.452637Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:11:04.452677Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:11:04.452690Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-01T06:11:04.452694Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-11-01T06:11:04.324Z] "POST /v1/api-keys HTTP/1.1" 400 - via_upstream - "-" 80 94 127 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "e36b7f3a-3c3f-4d96-933a-b3eee8f14df5" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.56:49290 10.133.1.56:443 10.132.0.14:43258 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T06:11:34.001521Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-01T06:11:42.432Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 108118 108394 350 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "cfac7de3-3140-413c-b5ce-23311d4a6b89" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:59060 10.133.1.56:443 10.132.0.14:57108 - v1-chat-completions-path [2026-11-01T06:11:43.446Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 108118 108394 333 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "c925b564-fe19-42b9-b510-e9aa3bbe8664" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.134.0.38:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:59066 10.133.1.56:443 10.132.0.14:57116 - v1-chat-completions-path 2026-11-01T06:11:58.217839Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:12:02.755462Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:12:05.411852Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T06:12:05.411890Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T06:12:05.411905Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-01T06:12:05.411910Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-01T06:12:05.295Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 324 116 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "57cccffa-fff6-474b-8b56-5a2bb21f9a7b" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.56:44416 10.133.1.56:443 10.132.0.14:44932 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T06:12:12.516186Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:12:15.363663Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:12:15.363705Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:12:15.363720Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:12:15.363726Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-01T06:12:15.645514Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-01T06:12:15.247Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 317 116 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "2546d7ce-72c4-4b33-b694-b91314fc81c3" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.56:34814 10.133.1.56:443 10.132.0.14:44304 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-01T06:12:15.565Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 79 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "d6cc077e-7613-412c-bfda-055d07c358dc" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:37148 10.133.1.56:443 10.132.0.14:44330 - v1-catch-all-path 2026-11-01T06:12:26.104247Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:12:35.281171Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:12:37.920765Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:12:37.920801Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:12:37.920816Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-01T06:12:37.920820Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-11-01T06:12:37.787Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 73 316 133 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "e9213d2d-4940-439c-94f9-af689a48c4c8" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.56:34814 10.133.1.56:443 10.132.0.14:34474 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T06:12:51.886200Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:13:03.457428Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:13:06.590348Z info ads ADS: new connection for node:3 2026-11-01T06:13:06.590528Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T06:13:06.590556Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-01T06:13:10.247983Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:13:49.406578Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:13:54.333588Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-01T06:13:54.275Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 333 87 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "a64b9db4-b56e-4b44-a542-4dae1b686bbb" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.1.64:8000" outbound|54321||e2e-prio-pool-25304531-inference-pool-ip-eb3771a1.llm.svc.cluster.local 10.133.1.56:44866 10.133.1.56:443 10.132.0.14:46036 - v1-model-routing 2026-11-01T06:14:13.274039Z info ads ADS: "" 3 terminated 2026-11-01T06:14:13.326857Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:14:18.150971Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:14:32.869744Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:14:44.703402Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:15:11.445799Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:15:36.555386Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:15:51.811224Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:16:00.260435Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:16:15.705286Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:16:20.488426Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:16:38.707388Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-01T06:16:41.690Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 283 78 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "73adc66d-e0e9-41a5-9b3f-8daf9ddfe08f" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:60660 10.133.1.56:443 10.132.0.14:38688 - v1-completions-path 2026-11-01T06:16:55.884809Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 [2026-11-01T06:16:55.820Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 70 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "65222c56-2809-45f1-aff9-b08bb6123cab" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:57428 10.133.1.56:443 10.132.0.14:53484 - v1-completions-path 2026-11-01T06:17:27.600441Z info ads ADS: new connection for node:4 2026-11-01T06:17:27.604442Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-01T06:17:27.605077Z info cache read certificate from file resource=file-root:system 2026-11-01T06:17:31.199862Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:17:32.594787Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:17:40.174974Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"149","StreamId":"8413858925991730678"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=26 [2026-11-01T06:17:40.110Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 65 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "a177ee3e-6ac7-465e-af6f-ea09d00734cc" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.1.56:443 10.132.0.14:33246 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-01T06:19:10.362622Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:19:10.362677Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:19:10.362701Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:19:10.362708Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T06:19:10.218Z] "DELETE /v1/api-keys/bea41ecb-1757-40e4-8afc-365945b9d1f8 HTTP/1.1" 200 - via_upstream - "-" 0 359 144 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "f6ebda0f-4a3e-4b2a-883c-842a9898ae8f" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.56:40828 10.133.1.56:443 10.132.0.14:34142 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T06:19:23.731021Z info ads ADS: "" 4 terminated 2026-11-01T06:19:23.773391Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:19:26.327737Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:19:28.114978Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T06:19:28.115066Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T06:19:28.115095Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-01T06:19:28.115120Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-01T06:19:27.998Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 117 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "2b7f3e47-6229-46c1-9272-22efd124e0e6" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.56:443 10.132.0.14:55154 - v1-completions-path 2026-11-01T06:19:36.563849Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:19:54.869521Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:20:11.757789Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-01T06:20:23.377Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 73 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "31a9ce37-3389-4ade-9ef4-27d8df1f3197" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:40934 10.133.1.56:443 10.132.0.14:35554 - v1-completions-path 2026-11-01T06:20:42.465320Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:20:53.026211Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:20:59.454385Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:21:10.105112Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:21:10.105150Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:21:10.105165Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:21:10.105171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T06:21:09.988Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 75 317 116 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "065cdf36-9d90-46c2-ba96-c17988ead474" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.56:34200 10.133.1.56:443 10.132.0.14:45052 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-01T06:21:10.851Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 67 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "3fcded0d-b0ce-4c21-8201-03599edde9d8" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:40934 10.133.1.56:443 10.132.0.14:45066 - v1-completions-path [2026-11-01T06:21:12.159Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 57 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "5cdd6691-0835-4476-a43f-de60c23638bb" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:51816 10.133.1.56:443 10.132.0.14:45072 - v1-completions-path [2026-11-01T06:21:13.208Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "0ce43fe2-10da-4997-b98a-e799928bf36e" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:51816 10.133.1.56:443 10.132.0.14:45074 - v1-completions-path 2026-11-01T06:21:28.072647Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:21:32.648327Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:21:45.931437Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-01T06:21:50.995Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 66 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "563c065c-e66c-4145-9515-98407d9c1495" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:57428 10.133.1.56:443 10.132.0.14:37092 - v1-completions-path 2026-11-01T06:21:55.897063Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:22:19.589349Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:22:30.190154Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:22:48.184898Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:22:50.863531Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:22:50.863568Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:22:50.863583Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-01T06:22:50.863588Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-01T06:22:50.745Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 322 117 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "93ef0d27-fb5d-430e-a77b-ead1ef8db627" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.56:58526 10.133.1.56:443 10.132.0.14:32936 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T06:22:54.317944Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:23:12.478476Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:23:17.905807Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:23:29.204802Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:23:41.429566Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T06:23:41.429614Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T06:23:41.429630Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-01T06:23:41.429635Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-01T06:23:41.316Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 310 113 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "7ba28240-b084-40b1-b880-f256aa1b0401" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.132.0.51:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.56:50874 10.133.1.56:443 10.132.0.14:34664 - odh-ai-gateway-infra.maas-api-route.3 2026-11-01T06:23:55.982136Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:24:06.654910Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:24:54.865922Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:25:02.472977Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:27:16.541977Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:28:45.022048Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:29:04.516054Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:29:09.504212Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:29:21.328492Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:29:38.131821Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:29:43.040195Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-01T06:29:46.481Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "4459234f-bcb7-47b3-8952-594e6884d716" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:51816 10.133.1.56:443 10.132.0.14:34418 - v1-completions-path 2026-11-01T06:30:02.726769Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:30:13.371534Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:30:19.910500Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-01T06:30:34.634Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 67 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "4af3c6da-d881-438f-b2b4-ad1453dac7cc" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:40934 10.133.1.56:443 10.132.0.14:35558 - v1-completions-path 2026-11-01T06:30:48.829263Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:30:55.288776Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:31:01.199229Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-01T06:31:12.097Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 60 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "edbf1173-336d-4333-bf99-489818a9b347" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.56:443 10.132.0.14:35008 - v1-completions-path [2026-11-01T06:31:12.359Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 62 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "50508d85-32c5-4359-9db5-1d671d829280" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:40934 10.133.1.56:443 10.132.0.14:35014 - v1-completions-path [2026-11-01T06:31:15.214Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "0ef06f3f-cbfb-4143-aeac-b21307dcae28" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:51816 10.133.1.56:443 10.132.0.14:35024 - v1-completions-path 2026-11-01T06:31:33.390376Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:31:44.116215Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:31:50.708582Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:31:56.540842Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:32:08.559887Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 [2026-11-01T06:32:15.622Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 70 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "761a778c-b93c-4330-a965-f4674881cdf8" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:40934 10.133.1.56:443 10.132.0.14:46408 - v1-completions-path [2026-11-01T06:32:17.100Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 67 - "44.212.242.249,10.132.0.14" "python-requests/2.33.0" "fe454a4a-0fc3-4055-8cc9-ee2280963eb0" "maas.apps.eda98aafc97c4de12f32.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.56:51816 10.133.1.56:443 10.132.0.14:46416 - v1-completions-path 2026-11-01T06:32:32.211979Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:32:42.873425Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:32:47.643464Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:33:05.843323Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:33:11.924930Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:33:35.590975Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:33:57.643667Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:34:17.545319Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:34:28.424839Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:34:39.342699Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:34:50.280923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:35:00.995918Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:35:11.820411Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:35:22.553113Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7 2026-11-01T06:35:33.171187Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=7