2026-11-04T01:55:28.649166Z info FLAG: --concurrency="0" 2026-11-04T01:55:28.649186Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-04T01:55:28.649191Z info FLAG: --help="false" 2026-11-04T01:55:28.649193Z info FLAG: --log_as_json="false" 2026-11-04T01:55:28.649195Z info FLAG: --log_caller="" 2026-11-04T01:55:28.649197Z info FLAG: --log_output_level="default:info" 2026-11-04T01:55:28.649200Z info FLAG: --log_stacktrace_level="default:none" 2026-11-04T01:55:28.649204Z info FLAG: --log_target="[stdout]" 2026-11-04T01:55:28.649207Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-04T01:55:28.649209Z info FLAG: --outlierLogPath="" 2026-11-04T01:55:28.649211Z info FLAG: --profiling="true" 2026-11-04T01:55:28.649213Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-04T01:55:28.649215Z info FLAG: --proxyLogLevel="warning" 2026-11-04T01:55:28.649217Z info FLAG: --serviceCluster="istio-proxy" 2026-11-04T01:55:28.649220Z info FLAG: --stsPort="0" 2026-11-04T01:55:28.649222Z info FLAG: --templateFile="" 2026-11-04T01:55:28.649224Z info FLAG: --tokenManagerPlugin="" 2026-11-04T01:55:28.649227Z info FLAG: --vklog="0" 2026-11-04T01:55:28.649229Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-04T01:55:28.649396Z info Proxy role ips=[10.134.0.44] type=router id=maas-default-gateway-openshift-default-58847f9fc5-nhhrz.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-04T01:55:28.649499Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-04T01:55:28.650966Z info cpu limit detected as 2, setting concurrency 2026-11-04T01:55:28.651290Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-04T01:55:28.651303Z info JWT policy is third-party-jwt 2026-11-04T01:55:28.651306Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-04T01:55:28.653230Z info Starting default Istio SDS Server 2026-11-04T01:55:28.653305Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-04T01:55:28.653340Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-04T01:55:28.654193Z info Opening status port 15020 2026-11-04T01:55:28.654923Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-04T01:55:28.656558Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-04T01:55:28.657614Z info Starting proxy agent 2026-11-04T01:55:28.657701Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-04T01:55:28.674080Z info sds Starting SDS grpc server 2026-11-04T01:55:28.674190Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-04T01:55:28.741192Z info cache generated new workload certificate resourceName=default latency=84.61294ms ttl=23h59m59.258813298s 2026-11-04T01:55:28.741233Z info cache Root cert has changed, start rotating root cert 2026-11-04T01:55:28.741281Z info cache returned workload trust anchor from cache ttl=23h59m59.258721183s 2026-11-04T01:55:28.764757Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-04T01:55:28.923017Z info ads ADS: new connection for node:1 2026-11-04T01:55:28.923106Z info cache returned workload certificate from cache ttl=23h59m59.076897292s 2026-11-04T01:55:28.923884Z info ads ADS: new connection for node:2 2026-11-04T01:55:28.924043Z info cache returned workload trust anchor from cache ttl=23h59m59.075959031s 2026-11-04T01:55:29.022151Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:55:37.715305Z info Readiness succeeded in 9.068996214s 2026-11-04T01:55:37.715716Z info Envoy proxy is ready 2026-11-04T01:58:04.921723Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:04.921764Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:04.921781Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:58:04.921787Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-04T01:58:04.832Z] "DELETE /v1/api-keys/af121704-e66e-43c6-b672-06ecd166e452 HTTP/1.1" 200 - via_upstream - "-" 0 321 88 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "340cdfaa-5b00-4cfd-8350-e9bfd80d9e1f" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:59950 10.134.0.44:443 10.132.0.15:44994 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:58:05.227173Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.227205Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.227216Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.227221Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:58:05.611416Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.611446Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.611458Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.611462Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T01:58:05.154Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 72 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "94cc9ff8-cdda-4b09-a0df-937de65dd284" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:59950 10.134.0.44:443 10.132.0.15:45008 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:58:05.347Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 52 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "51888b5f-55e3-49e4-abdb-e26350417520" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.44:443 10.132.0.15:45012 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:58:05.519Z] "DELETE /v1/api-keys/4f4ff296-7ffa-4f3a-9f19-a3986fa36a23 HTTP/1.1" 200 - via_upstream - "-" 0 321 91 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "5e6718f4-b6ef-41f3-a070-98ac61d1178a" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:59950 10.134.0.44:443 10.132.0.15:45024 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:58:05.899856Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.899906Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:05.899919Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:58:05.899923Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:06.189016Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-04T01:58:06.189099Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:06.189122Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:06.189140Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:58:06.189146Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:06.421940Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:06.421971Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:06.421984Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:06.421989Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:58:06.593453Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:06.593483Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:06.593495Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T01:58:06.593500Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:06.632071Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-04T01:58:06.632095Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-04T01:58:06.632097Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-04T01:58:06.632101Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-04T01:58:05.744Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 54 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "9a09b66a-6a32-4a47-b2d2-32be3e5c53a7" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.44:443 10.132.0.15:45028 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:58:05.821Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 78 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "cb143061-a199-49a4-8841-87374e6370a9" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:59950 10.134.0.44:443 10.132.0.15:45032 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:58:06.116Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 72 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "06ae4baa-9dbb-463d-a9aa-76a8f4dbaff4" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:59950 10.134.0.44:443 10.132.0.15:45042 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:58:06.355Z] "DELETE /v1/api-keys/e6ca806f-d233-4a83-828a-a6d5a63581f8 HTTP/1.1" 200 - via_upstream - "-" 0 321 66 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "7b508cb7-12db-4cbe-adaa-2ea91af786e7" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:59950 10.134.0.44:443 10.132.0.15:45056 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:58:06.513Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 79 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "2624a7de-c21c-4adb-9730-651dca0558b2" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:59950 10.134.0.44:443 10.132.0.15:45068 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T01:58:06.592Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 77 40 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "ff447807-0931-49dc-b747-6a4b2f91f305" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.44:443 10.132.0.15:45080 - v1-completions-path [2026-11-04T01:58:07.489Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 278 84 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "631c6faa-cc07-4b58-ab28-b3e635565c26" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.133.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.44:55626 10.134.0.44:443 10.132.0.15:45086 - v1-completions-path 2026-11-04T01:58:08.417313Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:08.417348Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:08.417363Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:58:08.417368Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T01:58:08.087Z] "POST /maas-api/internal/v1/api-keys/validate HTTP/1.1" 404 NR route_not_found - "-" 31 0 7 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "0819807f-1870-4326-a70c-1f36bdf6e7d2" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "-" - - 10.134.0.44:443 10.132.0.15:45098 - - [2026-11-04T01:58:08.351Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 65 - "34.228.250.194,10.132.0.15" "python-requests/2.34.2" "fa054b46-c5cc-4489-bc5b-d424bbaf0ccf" "maas.apps.b5b84be2b10d47dc01de.prod.konflux-ocp-ci.dev" "10.134.0.33:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.44:59950 10.134.0.44:443 10.132.0.15:45110 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:59:12.683773Z info ads ADS: new connection for node:3 2026-11-04T01:59:12.701340Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-04T01:59:12.701395Z info cache read certificate from file resource=file-root:system 2026-11-04T01:59:13.493317Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:59:31.698870Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:00:22.055015Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:01:06.228233Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:01:27.623924Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:01:32.580163Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:02:26.122953Z info ads ADS: "" 3 terminated with stream closed 2026-11-04T02:02:40.660847Z info ads ADS: new connection for node:4 2026-11-04T02:02:40.672845Z info cache read certificate from file resource=file-root:system 2026-11-04T02:02:42.095683Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:02:50.797978Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:03:15.493661Z info ads ADS: "" 4 terminated with stream closed 2026-11-04T02:03:15.565940Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:03:15.860326Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:03:26.980873Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8