2026-11-04T02:03:36.317649Z info FLAG: --concurrency="0" 2026-11-04T02:03:36.317668Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-04T02:03:36.317672Z info FLAG: --help="false" 2026-11-04T02:03:36.317675Z info FLAG: --log_as_json="false" 2026-11-04T02:03:36.317677Z info FLAG: --log_caller="" 2026-11-04T02:03:36.317679Z info FLAG: --log_output_level="default:info" 2026-11-04T02:03:36.317681Z info FLAG: --log_stacktrace_level="default:none" 2026-11-04T02:03:36.317685Z info FLAG: --log_target="[stdout]" 2026-11-04T02:03:36.317687Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-04T02:03:36.317689Z info FLAG: --outlierLogPath="" 2026-11-04T02:03:36.317691Z info FLAG: --profiling="true" 2026-11-04T02:03:36.317693Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-04T02:03:36.317696Z info FLAG: --proxyLogLevel="warning" 2026-11-04T02:03:36.317697Z info FLAG: --serviceCluster="istio-proxy" 2026-11-04T02:03:36.317699Z info FLAG: --stsPort="0" 2026-11-04T02:03:36.317701Z info FLAG: --templateFile="" 2026-11-04T02:03:36.317703Z info FLAG: --tokenManagerPlugin="" 2026-11-04T02:03:36.317706Z info FLAG: --vklog="0" 2026-11-04T02:03:36.317708Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-04T02:03:36.317862Z info Proxy role ips=[10.134.0.33] type=router id=maas-default-gateway-openshift-default-7bdc57c4cf-r74jj.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-04T02:03:36.317912Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-04T02:03:36.318905Z info cpu limit detected as 2, setting concurrency 2026-11-04T02:03:36.319407Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-04T02:03:36.319425Z info JWT policy is third-party-jwt 2026-11-04T02:03:36.319429Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-04T02:03:36.321760Z info Starting default Istio SDS Server 2026-11-04T02:03:36.321743Z info Opening status port 15020 2026-11-04T02:03:36.321784Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-04T02:03:36.321808Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-04T02:03:36.322658Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-04T02:03:36.324418Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-04T02:03:36.325232Z info Starting proxy agent 2026-11-04T02:03:36.325264Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-04T02:03:36.332445Z info sds Starting SDS grpc server 2026-11-04T02:03:36.332465Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-04T02:03:36.399593Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-04T02:03:36.530471Z info cache generated new workload certificate resourceName=default latency=207.592954ms ttl=23h59m59.469534099s 2026-11-04T02:03:36.530523Z info cache Root cert has changed, start rotating root cert 2026-11-04T02:03:36.530558Z info cache returned workload trust anchor from cache ttl=23h59m59.469442411s 2026-11-04T02:03:36.537590Z info ads ADS: new connection for node:1 2026-11-04T02:03:36.537674Z info cache returned workload certificate from cache ttl=23h59m59.46232879s 2026-11-04T02:03:36.538625Z info ads ADS: new connection for node:2 2026-11-04T02:03:36.538850Z info cache returned workload trust anchor from cache ttl=23h59m59.461151765s 2026-11-04T02:03:36.583059Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:03:42.436925Z info Readiness succeeded in 6.121553275s 2026-11-04T02:03:42.437272Z info Envoy proxy is ready 2026-11-04T02:05:47.626470Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:05:50.860193Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:06:43.936017Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:06:52.814007Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:06:58.722154Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:07:04.630034Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:07:14.264902Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:07:22.465090Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-04T02:08:02.026Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 67 - "34.228.250.194,10.132.0.20" "curl/7.76.1" "bcacab96-973c-458b-ac65-fbe3a95065b5" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:48890 - v1-model-routing 2026-11-04T02:08:02.596941Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-04T02:08:02.596972Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-04T02:08:02.596976Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-04T02:08:02.596980Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-04T02:08:02.316Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 56 - "34.228.250.194,10.132.0.20" "curl/7.76.1" "15bd4cda-ddf1-4a41-90d5-5beaa79eefea" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:48900 - v1-model-routing [2026-11-04T02:08:02.579Z] "POST /v1/chat/completions HTTP/1.1" 401 - - "-" 122 88 17 - "34.228.250.194,10.132.0.20" "curl/7.76.1" "7f5d2e4e-21e1-4593-9d50-3ac11f816fbf" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:48902 - v1-model-routing 2026-11-04T02:12:42.462763Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:42.462796Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:42.462809Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:42.462816Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-04T02:12:42.348Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 114 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "2314ce04-4fbc-4291-bcd8-fdf7431bff08" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38784 10.134.0.33:443 10.132.0.20:39372 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:12:42.786340Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:42.786373Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:42.786384Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:42.786389Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T02:12:42.668Z] "DELETE /v1/api-keys/69f5e526-16b3-48d1-adfa-a298fd71e96e HTTP/1.1" 200 - via_upstream - "-" 0 321 118 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "965f1b04-96e1-4041-83b3-8e62c4ffb214" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38792 10.134.0.33:443 10.132.0.20:39386 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:12:51.065212Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:51.065245Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:51.065257Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T02:12:51.065262Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:51.375160Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:51.375196Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:51.375212Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T02:12:51.375218Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T02:12:50.957Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 107 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "39e45cca-3214-4475-9cbd-1a9c4853f75a" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38792 10.134.0.33:443 10.132.0.20:39402 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T02:12:51.267Z] "DELETE /v1/api-keys/1d1ee8b6-6e7e-48a1-81af-b26a000d1a26 HTTP/1.1" 200 - via_upstream - "-" 0 321 107 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "7fc21ae5-41bc-4c93-bba6-a0e94cf320d8" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38784 10.134.0.33:443 10.132.0.20:44820 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:12:52.874710Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:52.874747Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:52.874764Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:52.874770Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T02:12:52.769Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 104 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "6a239131-0781-4098-80a6-d9a87682eb47" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38792 10.134.0.33:443 10.132.0.20:44842 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T02:12:53.900Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 84 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "dadb1c2c-f36e-4faf-b494-ef25657d1287" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.33:443 10.132.0.20:44850 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:12:55.066662Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:55.066699Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:55.066714Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:55.066720Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-04T02:12:54.960Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 105 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "71161a97-0cce-4ddf-bd27-493a1f412571" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38784 10.134.0.33:443 10.132.0.20:44852 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T02:12:55.578Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 85 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "568ac343-ba48-47c3-ae1a-5189d935557e" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.33:443 10.132.0.20:44856 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:12:55.970289Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-04T02:12:55.970360Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:55.970379Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:55.970390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T02:12:55.970395Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:56.326612Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:56.326643Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:12:56.326655Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T02:12:56.326659Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-04T02:12:55.862Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 107 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "9c143cfb-8c9b-40de-9e54-2b483edfa996" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38792 10.134.0.33:443 10.132.0.20:44868 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T02:12:56.222Z] "DELETE /v1/api-keys/9cbaaf7f-dc69-4afe-b1ff-4dea2cb203f4 HTTP/1.1" 200 - via_upstream - "-" 0 321 104 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "8446308a-4644-4c0c-a77a-c055e62d4704" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38792 10.134.0.33:443 10.132.0.20:44876 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:12:56.834740Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:56.834771Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:56.834783Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:12:56.834787Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-04T02:12:56.729Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 105 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "cae2d648-8789-4f70-9262-9b4aaec05f04" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38784 10.134.0.33:443 10.132.0.20:44892 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T02:12:57.751Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 279 79 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "1f93c666-ffd8-49e9-93b1-d861309dc7c3" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.134.0.58:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.33:36580 10.134.0.33:443 10.132.0.20:44900 - v1-completions-path [2026-11-04T02:12:58.298Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 287 61 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "2f0f3c59-07a3-4318-88f0-36ace7066353" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.134.0.58:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.33:36580 10.134.0.33:443 10.132.0.20:44912 - v1-completions-path [2026-11-04T02:12:59.007Z] "POST /maas-api/internal/v1/subscriptions/select HTTP/1.1" 404 NR route_not_found - "-" 56 0 7 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "714f1a48-ccda-41c4-9ef8-35064359db33" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" - - 10.134.0.33:443 10.132.0.20:44924 - - 2026-11-04T02:13:02.640249Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:13:02.640284Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:13:02.640296Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:13:02.640301Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T02:13:02.528Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 93 316 111 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "b5123a6b-14e5-40ce-b132-d73527021444" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38792 10.134.0.33:443 10.132.0.20:55600 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:13:08.270846Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:13:08.270882Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:13:08.270894Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:13:08.270899Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T02:13:08.262Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 8 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "20398e9d-6a76-487e-994e-bc9c2f464dad" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:38792 10.134.0.33:443 10.132.0.20:55612 - odh-ai-gateway-infra.maas-api-route.4 2026-11-04T02:15:01.002977Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-04T02:15:00.996Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 83 0 43 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "cb7795ec-81d4-439e-abde-e4cae0575ee3" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.134.0.33:443 10.132.0.20:46900 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:16:14.005390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:16:14.005427Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:16:14.005440Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T02:16:14.005445Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:16:14.338188Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:16:14.338225Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:16:14.338238Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:16:14.338243Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T02:16:13.876Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 129 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "91ed174d-0235-493b-bc6b-e0894dd15ad1" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:55790 10.134.0.33:443 10.132.0.20:38060 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T02:16:14.223Z] "DELETE /v1/api-keys/d2526206-20ff-43a1-8f46-71dac5cc9793 HTTP/1.1" 200 - via_upstream - "-" 0 321 114 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "f2e82db2-abf3-4c3e-902f-8bdd927edc32" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:55796 10.134.0.33:443 10.132.0.20:38086 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:16:14.965192Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:16:14.965226Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:16:14.965237Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:16:14.965242Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T02:16:15.280686Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:16:15.280732Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:16:15.280752Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T02:16:15.280759Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T02:16:14.848Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 116 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "81d4f042-0d0b-4237-8c57-d25a85f984c0" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:55796 10.134.0.33:443 10.132.0.20:38090 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T02:16:15.170Z] "DELETE /v1/api-keys/93820cb8-72fc-4f6d-beb8-36a45fc3abc4 HTTP/1.1" 200 - via_upstream - "-" 0 321 110 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "e89cee96-03dc-41d0-8c2f-6f7c36e07a56" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "10.133.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.33:55790 10.134.0.33:443 10.132.0.20:38096 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:16:17.352936Z info ads ADS: new connection for node:3 2026-11-04T02:16:17.359421Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-04T02:16:17.359535Z info cache read certificate from file resource=file-root:system 2026-11-04T02:16:22.296381Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:16:54.659481Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:17:06.453154Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:17:06.453190Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:17:06.453205Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T02:17:06.453211Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:17:06.453400Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-04T02:17:06.324Z] "POST /anything/e2e-dft-4307ea-5a04cd HTTP/1.1" 200 - via_upstream - "-" 109 1596 129 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "5690b527-5e66-40a7-b9d2-b8c59bbf2c9f" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.33:59508 10.134.0.33:443 10.132.0.20:50762 - llm.external-model-e2e-dft-4307ea-7e507f17f6dc.0 2026-11-04T02:17:32.555100Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-04T02:18:10.006Z] "POST /llm/e2e-dft-a60858/anything/e2e-dft-a60858-12c938 HTTP/1.1" 503 - - "-" 109 0 79 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "00af9c83-431c-471d-a72e-d2f8c49e3819" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-a60858.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:45644 - llm.external-model-e2e-dft-a60858-325448bfc54b.2 [2026-11-04T02:18:20.907Z] "POST /llm/e2e-dft-a60858/anything/e2e-dft-a60858-12c938 HTTP/1.1" 503 - - "-" 109 0 68 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "4384b640-f989-4bdf-8893-b3bcb2feac29" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-a60858.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:38058 - llm.external-model-e2e-dft-a60858-325448bfc54b.2 [2026-11-04T02:21:26.790Z] "POST /llm/e2e-dft-a60858/anything/e2e-dft-a60858-12c938 HTTP/1.1" 503 - - "-" 109 0 74 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "fe3b2f74-9201-4bfb-ad33-98572d9b13f7" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-a60858.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:34172 - llm.external-model-e2e-dft-a60858-325448bfc54b.2 2026-11-04T02:21:41.976537Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:21:53.358384Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:21:53.358427Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:21:53.358443Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T02:21:53.358450Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:21:53.358740Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-04T02:21:53.203Z] "POST /anything/e2e-dft-a60858-12c938 HTTP/1.1" 200 - via_upstream - "-" 109 1598 155 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "5f8f128d-4dad-4350-aa80-517bc3f031ea" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.33:53606 10.134.0.33:443 10.132.0.20:38018 - llm.external-model-e2e-dft-a60858-325448bfc54b.0 2026-11-04T02:22:01.475169Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-04T02:22:08.579Z] "POST /llm/e2e-dft-9aa046/anything/e2e-dft-9aa046-c0a1dc HTTP/1.1" 503 - - "-" 109 0 82 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "90d938e1-40cc-4c6e-9fdf-d1ec45d4cca4" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-9aa046.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:48106 - llm.external-model-e2e-dft-9aa046-6e39e6aff2e1.2 2026-11-04T02:22:25.090168Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-04T02:23:12.393Z] "POST /llm/e2e-dft-9aa046/anything/e2e-dft-9aa046-c0a1dc HTTP/1.1" 503 - - "-" 109 0 87 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "6fb84222-a4c9-4edf-ad35-7c6c6d30f490" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-9aa046.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:49238 - llm.external-model-e2e-dft-9aa046-6e39e6aff2e1.2 [2026-11-04T02:23:17.854Z] "POST /llm/e2e-dft-9aa046/anything/e2e-dft-9aa046-c0a1dc HTTP/1.1" 503 - - "-" 109 0 71 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "6b7d43c2-9f90-442b-aded-644eb0c03d8f" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-9aa046.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:49254 - llm.external-model-e2e-dft-9aa046-6e39e6aff2e1.2 [2026-11-04T02:23:23.129Z] "POST /llm/e2e-dft-9aa046/anything/e2e-dft-9aa046-c0a1dc HTTP/1.1" 503 - - "-" 109 0 63 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "f04b483a-0229-415c-9573-0f298d5da1b2" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-9aa046.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:56396 - llm.external-model-e2e-dft-9aa046-6e39e6aff2e1.2 [2026-11-04T02:23:28.395Z] "POST /llm/e2e-dft-9aa046/anything/e2e-dft-9aa046-c0a1dc HTTP/1.1" 503 - - "-" 109 0 71 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "e2b68ece-f4be-468b-abdd-be324f983411" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-9aa046.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:56408 - llm.external-model-e2e-dft-9aa046-6e39e6aff2e1.2 [2026-11-04T02:24:00.279Z] "POST /llm/e2e-dft-9aa046/anything/e2e-dft-9aa046-c0a1dc HTTP/1.1" 503 - - "-" 109 0 67 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "66f58f91-6bbe-4ff0-a4e9-88c90c249837" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-9aa046.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:44832 - llm.external-model-e2e-dft-9aa046-6e39e6aff2e1.2 [2026-11-04T02:24:53.603Z] "POST /llm/e2e-dft-9aa046/anything/e2e-dft-9aa046-c0a1dc HTTP/1.1" 503 - - "-" 109 0 66 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "cecfc2a8-fefd-4bfc-bab5-3726aaf4409c" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-9aa046.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:41888 - llm.external-model-e2e-dft-9aa046-6e39e6aff2e1.2 [2026-11-04T02:26:23.671Z] "POST /llm/e2e-dft-9aa046/anything/e2e-dft-9aa046-c0a1dc HTTP/1.1" 503 - - "-" 109 0 69 - "34.228.250.194,10.132.0.20" "python-requests/2.34.2" "29e3c7d1-454f-4d17-880b-f7e46325e9dd" "maas.apps.e038e5fb261d0e85f615.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-9aa046.llm.svc.cluster.local - 10.134.0.33:443 10.132.0.20:55624 - llm.external-model-e2e-dft-9aa046-6e39e6aff2e1.2 2026-11-04T02:27:35.641265Z info ads ADS: "" 3 terminated 2026-11-04T02:27:39.188181Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-04T02:27:41.348592Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9