2026-11-02T22:13:13.286541Z info FLAG: --concurrency="0" 2026-11-02T22:13:13.286567Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T22:13:13.286572Z info FLAG: --help="false" 2026-11-02T22:13:13.286574Z info FLAG: --log_as_json="false" 2026-11-02T22:13:13.286576Z info FLAG: --log_caller="" 2026-11-02T22:13:13.286578Z info FLAG: --log_output_level="default:info" 2026-11-02T22:13:13.286580Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T22:13:13.286584Z info FLAG: --log_target="[stdout]" 2026-11-02T22:13:13.286587Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T22:13:13.286588Z info FLAG: --outlierLogPath="" 2026-11-02T22:13:13.286590Z info FLAG: --profiling="true" 2026-11-02T22:13:13.286593Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T22:13:13.286595Z info FLAG: --proxyLogLevel="warning" 2026-11-02T22:13:13.286597Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T22:13:13.286599Z info FLAG: --stsPort="0" 2026-11-02T22:13:13.286601Z info FLAG: --templateFile="" 2026-11-02T22:13:13.286603Z info FLAG: --tokenManagerPlugin="" 2026-11-02T22:13:13.286605Z info FLAG: --vklog="0" 2026-11-02T22:13:13.286608Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T22:13:13.286747Z info Proxy role ips=[10.133.1.47] type=router id=maas-default-gateway-openshift-default-77bfb58f8c-hft2d.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T22:13:13.286795Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T22:13:13.287957Z info cpu limit detected as 2, setting concurrency 2026-11-02T22:13:13.288231Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T22:13:13.288243Z info JWT policy is third-party-jwt 2026-11-02T22:13:13.288246Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T22:13:13.289904Z info Starting default Istio SDS Server 2026-11-02T22:13:13.289926Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T22:13:13.289948Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T22:13:13.290325Z info Opening status port 15020 2026-11-02T22:13:13.291169Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T22:13:13.292478Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T22:13:13.293378Z info Starting proxy agent 2026-11-02T22:13:13.293427Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T22:13:13.302987Z info sds Starting SDS grpc server 2026-11-02T22:13:13.303005Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T22:13:13.355500Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T22:13:13.513449Z info ads ADS: new connection for node:1 2026-11-02T22:13:13.513893Z info ads ADS: new connection for node:2 2026-11-02T22:13:13.597275Z info cache generated new workload certificate resourceName=default latency=306.279217ms ttl=23h59m59.402728685s 2026-11-02T22:13:13.597306Z info cache Root cert has changed, start rotating root cert 2026-11-02T22:13:13.597340Z info cache returned workload trust anchor from cache ttl=23h59m59.402660312s 2026-11-02T22:13:13.597337Z info cache returned workload certificate from cache ttl=23h59m59.402668128s 2026-11-02T22:13:13.597423Z info cache returned workload trust anchor from cache ttl=23h59m59.402577707s 2026-11-02T22:13:13.597740Z info cache returned workload trust anchor from cache ttl=23h59m59.402260557s 2026-11-02T22:13:13.696228Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:13:20.037479Z error failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused 2026-11-02T22:13:21.571033Z info Readiness succeeded in 8.287167069s 2026-11-02T22:13:21.571462Z info Envoy proxy is ready 2026-11-02T22:13:32.743593Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:13:43.569412Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:13:48.598888Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T22:14:00.139Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 82 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "6be9dbbf-ee55-46a7-acd6-d7a59e640c31" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.47:33898 10.133.1.47:443 10.132.0.12:39002 - v1-completions-path [2026-11-02T22:14:00.940Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 56 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "d03d8421-cef0-429d-af2f-e2af849e7dca" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.47:33898 10.133.1.47:443 10.132.0.12:39016 - v1-completions-path [2026-11-02T22:14:02.588Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 72 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "4d143e3c-0e7a-44ca-a20b-95766428a9e7" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "10.134.0.37:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.1.47:33904 10.133.1.47:443 10.132.0.12:39030 - v1-completions-path 2026-11-02T22:14:19.651415Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T22:14:28.572Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 74 292 74 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "d628a82f-a04d-4ce8-98ca-48aca956bf5b" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "10.133.0.45:8000" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local 10.133.1.47:43736 10.133.1.47:443 10.132.0.12:56564 - v1-completions-path 2026-11-02T22:14:42.192804Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:14:49.025221Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:15:15.386454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:15:34.809327Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-02T22:15:34.801Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 76 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "41ca75cf-a6e6-47d4-874e-8309526ca1e8" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.47:55774 10.133.1.47:443 10.132.0.12:34520 - v1-completions-path 2026-11-02T22:15:37.375704Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T22:16:03.962Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 39 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "72076c17-3f89-4ea9-b042-18d96d775617" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.47:443 10.132.0.12:48462 - v1-completions-path [2026-11-02T22:16:12.834Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 0 33 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "99ad8687-b0c7-489c-ac74-45aa7fc312bf" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.1.47:443 10.132.0.12:50840 - v1-completions-path 2026-11-02T22:16:20.956883Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:16:29.682749Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:16:29.682784Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:16:29.682800Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:16:29.682806Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T22:16:29.570Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 65 307 112 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "c141147d-ba8b-4db3-a798-16e9c72e2f4e" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "10.132.0.36:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.47:52238 10.133.1.47:443 10.132.0.12:39870 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:16:44.025480Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:16:54.672223Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:20:54.638888Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:21:11.787755Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:21:16.296052Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T22:21:26.741Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 279 69 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "e1b6638e-732b-4611-bdcf-8e84018b00d9" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "10.134.0.39:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.47:34992 10.133.1.47:443 10.132.0.12:46298 - v1-completions-path 2026-11-02T22:21:40.703028Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:23:33.935921Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:23:33.935956Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:23:33.935986Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:23:33.935992Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T22:23:33.871Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 64 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "267d39c3-c6e3-4e7d-afb7-9467003b22cd" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "10.132.0.36:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.1.47:36026 10.133.1.47:443 10.132.0.12:45556 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:23:42.159703Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:24:23.821981Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:24:50.176592Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:25:05.291777Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:25:13.549728Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:25:20.134760Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:25:31.144343Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:25:51.235149Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:29:26.307993Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:29:37.900089Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:29:44.711622Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-02T22:29:44.713188Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-02T22:29:44.636Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 76 - "34.228.250.194,10.132.0.12" "python-requests/2.34.2" "88a9ce4f-80ff-4a43-9881-4f99446bf6dc" "maas.apps.f3c7dd50665b28c8e686.prod.konflux-ocp-ci.dev" "10.134.0.40:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.1.47:35586 10.133.1.47:443 10.132.0.12:44258 - v1-catch-all-path 2026-11-02T22:29:59.015473Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:30:03.356209Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:30:19.650082Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T22:30:31.356201Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8