2026-05-25T02:59:04.369409Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:04.376284Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:04.376416Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:04.376526Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:04.378081Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:04.378138Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:04.378148Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:04.379553Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:04.379610Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T02:59:04.379621Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-25T02:59:04.368Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 9 - "10.132.0.53" "Go-http-client/1.1" "0180c6c7-474e-4304-a0c6-9a370df5ec79" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.28:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:38382 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-25T02:59:04.344Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 471 35 - "100.64.0.4" "python-requests/2.32.5" "a972bb03-d03a-494f-b5c5-30ed053648c4" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.4:9508 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T02:59:18.423713Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:18.423765Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:18.438937Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:18.439067Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:18.439288Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:18.443620Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:18.443678Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T02:59:18.443688Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:18.459818Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:18.459859Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:18.469726Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:18.469838Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:18.470035Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:18.473445Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:18.473511Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:18.473522Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-25T02:59:18.422Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 55 297 21 - "100.64.0.4" "python-requests/2.32.5" "b77a2489-1fd2-4d6c-a234-c6735e1cdfe2" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.4:15564 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T02:59:18.458Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 55 298 14 - "100.64.0.2" "python-requests/2.32.5" "a11fe2ea-d185-415d-a19b-e97daaa843ee" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.2:47604 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T02:59:26.502489Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.502540Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.506644Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.506765Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.509172Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.509203Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.515122Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.515238Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.515334Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.516790Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.516845Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:26.516855Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.518120Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.518168Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:26.518178Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.541981Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:26.542020Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:26.545388Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:26.545493Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:26.547583Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.547615Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.553261Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.553377Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.553468Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.555924Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.555972Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:26.555981Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:26.557190Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:26.557242Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T02:59:26.557254Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-25T02:59:26.508Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 8 - "10.132.0.53" "Go-http-client/1.1" "2204c9d4-eb2f-4818-886a-d015b4e0e4b8" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.28:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:38382 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-25T02:59:26.501Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 468 16 - "100.64.0.3" "python-requests/2.32.5" "eb5acf92-efb9-4f81-8168-f024b2476383" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.3:17424 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T02:59:26.546Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 166 9 - "10.132.0.53" "Go-http-client/1.1" "6183b08d-0aaa-44a2-81fc-499cf89eef5d" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.134.0.33:8000" outbound|8000||e2e-distinct-2-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:36992 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-distinct-2-simulated-kserve-route.2 [2026-05-25T02:59:26.540Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 345 16 - "100.64.0.3" "python-requests/2.32.5" "e7fe188e-bf34-4020-be0a-1faf2c513671" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.3:21768 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T02:59:42.932981Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:42.933027Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:42.946821Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:42.946970Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:42.949655Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.949691Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.958494Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.958618Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.960177Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.960253Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:42.960270Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.961839Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.961905Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.971003Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.971126Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.972720Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.972781Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:42.972797Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.974124Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.974160Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.981488Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.981603Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.983675Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.983731Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:42.983747Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:42.984970Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:42.985033Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T02:59:42.985045Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-25T02:59:42.948Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 11 - "10.132.0.53" "Go-http-client/1.1" "272a08b1-e540-4a05-8386-6f22b4bb2aee" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.28:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:38382 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-25T02:59:42.961Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 166 11 - "10.132.0.53" "Go-http-client/1.1" "32daebc5-30a6-4428-b73e-7d04db78ac53" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.134.0.33:8000" outbound|8000||e2e-distinct-2-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:36992 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-distinct-2-simulated-kserve-route.2 [2026-05-25T02:59:42.973Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 10 - "10.132.0.53" "Go-http-client/1.1" "3249b24c-ef69-476d-b6a1-38d02c45c908" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.30:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:47828 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-25T02:59:42.931Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 1361 53 - "100.64.0.4" "python-requests/2.32.5" "07190f18-d212-47c0-a61d-4173e61659ff" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.4:3048 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T02:59:59.370802Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:59.370851Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:59.384269Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:59.384394Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:59.386911Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.386943Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.395859Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.396001Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.397476Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.397531Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:59.397542Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.398865Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:59.398939Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T02:59:59.398951Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T02:59:59.418606Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.418643Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.427683Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.427784Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.429863Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.429924Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.437398Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.437504Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.439078Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.439131Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:59.439142Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.440268Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T02:59:59.440314Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T02:59:59.440324Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-25T02:59:59.385Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 11 - "10.132.0.53" "Go-http-client/1.1" "eb4ecddd-97ef-4fa3-8275-f6e371b295fb" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.28:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:38382 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-25T02:59:59.369Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 470 29 - "100.64.0.2" "python-requests/2.32.5" "a75996fc-d933-4e7c-a150-f283062c7ea1" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.2:2888 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T02:59:59.429Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 166 10 - "10.132.0.53" "Go-http-client/1.1" "7e5423fa-576c-4601-871d-c6d7a96689c3" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.134.0.33:8000" outbound|8000||e2e-distinct-2-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:36992 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-distinct-2-simulated-kserve-route.2 [2026-05-25T02:59:59.417Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 347 22 - "100.64.0.2" "python-requests/2.32.5" "c0c47ee3-3c08-4f6b-9583-32648500c4db" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.2:21188 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T03:00:08.436050Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:08.436099Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:08.437793Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:08.437828Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-25T03:00:08.434Z] "GET /maas-api/v1/models HTTP/1.1" 401 - - "-" 0 0 3 - "100.64.0.4" "python-requests/2.32.5" "7c2e6471-c200-4326-b04a-e19ebe8214ae" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.133.0.25:443 100.64.0.4:57854 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T03:00:25.044984Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.045031Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.061115Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.061244Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.061461Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.066961Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.067017Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:00:25.067028Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-25T03:00:25.043Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 330 23 - "100.64.0.3" "python-requests/2.32.5" "ff208dc5-6b72-4fcb-bda9-8d3ed61c3bca" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.3:39176 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T03:00:25.090980Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.091012Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.098655Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.098774Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.099050Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.099080Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.100159Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.100185Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.100397Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.102703Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.102841Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.103040Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.103477Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.121922Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.121955Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.128610Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.128729Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.128995Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.129023Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.129305Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.129326Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.129524Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.131269Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.131403Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.131578Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.131897Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.146899Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.146929Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.153951Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.154066Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.154287Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.154315Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.154564Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.154592Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:25.182958Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.182997Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.189697Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.189819Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.190138Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.190171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.190543Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.190567Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.210566Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.210607Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.225009Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.225141Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.228459Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.228498Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.235538Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.235659Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.235769Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.238756Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.238817Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:00:25.238827Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.240693Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:25.240749Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:00:25.240760Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-25T03:00:25.090Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 13 - "100.64.0.2" "python-requests/2.32.5" "46ae2353-c2b6-402d-8b46-fe517ab7643e" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:57496 10.133.0.25:443 100.64.0.2:34196 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.0 [2026-05-25T03:00:25.121Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 10 - "100.64.0.4" "python-requests/2.32.5" "2a616f76-b54e-41c9-94f2-5cd8cafe4185" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:57506 10.133.0.25:443 100.64.0.4:35126 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.0 [2026-05-25T03:00:25.146Z] "POST /v1/completions HTTP/1.1" 429 - - "-" 66 18 8 - "100.64.0.3" "python-requests/2.32.5" "d61d0f27-2e3c-4fb3-a65d-4444d4a169b1" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:57506 10.133.0.25:443 100.64.0.3:39188 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.0 [2026-05-25T03:00:25.182Z] "POST /v1/completions HTTP/1.1" 429 - - "-" 66 18 8 - "100.64.0.2" "python-requests/2.32.5" "5704bb47-bf03-4608-8c70-e5ce979814d0" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:57496 10.133.0.25:443 100.64.0.2:34208 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.0 [2026-05-25T03:00:25.227Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 11 - "10.132.0.53" "Go-http-client/1.1" "b7cd6603-e654-4f89-a280-bd160431d4a2" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:53670 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.2 [2026-05-25T03:00:25.209Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 547 31 - "100.64.0.4" "python-requests/2.32.5" "ce0d970f-0910-43b1-8195-e1da84ec9335" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.4:35134 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T03:00:34.131214Z info ads ADS: new connection for node:6 2026-05-25T03:00:34.133369Z info cache read certificate from file resource=file-root:system 2026-05-25T03:00:40.803641Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:40.803692Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:40.816561Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:40.816718Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:40.817005Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:40.821231Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:40.821292Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:00:40.821302Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-25T03:00:40.802Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 79 321 18 - "100.64.0.3" "python-requests/2.32.5" "f6933a89-11a7-4af7-8db9-347aee597cb4" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.3:33520 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T03:00:41.164341Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.164390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.165539Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.165574Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.183929Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:41.183966Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:41.184972Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:41.185004Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:00:41.202347Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.202382Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.208799Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.208948Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.209202Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.209233Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.210427Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.210460Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.210709Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.211748Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:00:41.211934Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:00:41.211948Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-25T03:00:41.163Z] "POST /llm/e2e-external-model/v1/chat/completions HTTP/1.1" 401 - - "-" 83 0 2 - "100.64.0.3" "python-requests/2.32.5" "3518cedc-d272-4759-91b5-a5e15668efd1" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "-" outbound|443||httpbin.org - 10.133.0.25:443 100.64.0.3:46238 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-external-model.0 [2026-05-25T03:00:41.182Z] "POST /llm/e2e-external-model/v1/chat/completions HTTP/1.1" 401 - - "-" 83 0 2 - "100.64.0.4" "python-requests/2.32.5" "e79a50ef-1d3e-4e8c-93b7-e8e24d4e94e8" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "-" outbound|443||httpbin.org - 10.133.0.25:443 100.64.0.4:38856 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-external-model.0 [2026-05-25T03:00:41.201Z] "POST /llm/e2e-external-model/v1/chat/completions HTTP/1.1" 404 - - "-" 83 98 10 - "100.64.0.2" "python-requests/2.32.5" "659431fd-fad6-4aec-952b-2547bdaa0f16" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "-" outbound|443||httpbin.org - 10.133.0.25:443 100.64.0.2:29098 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.e2e-external-model.0 [2026-05-25T03:00:56.959Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.133.0.25:80 100.64.0.3:41534 - - [2026-05-25T03:01:01.107Z] "GET / HTTP/1.1" 404 NR route_not_found - "-" 0 0 1 - "100.64.0.2" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:134.0) Gecko/20100101 Firefox/134.0" "adf2eb98-eec4-40d7-be71-8f7d8d2c81a9" "44.205.213.62" "-" - - 10.133.0.25:80 100.64.0.2:63328 - - 2026-05-25T03:01:18.807239Z info ads ADS: "" 6 terminated with stream closed 2026-05-25T03:01:20.555350Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.555401Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.558268Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.558395Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.558607Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.571745Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.571814Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:20.571825Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.659237Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:20.659293Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:20.684360Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:20.684403Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:20.703520Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.703569Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.705892Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.705930Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.723462Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.723507Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.725040Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:20.725077Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.068333Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.068390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.071334Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.071462Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.071675Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.075586Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.075642Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:21.075653Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-25T03:01:20.554Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 314 17 - "100.64.0.2" "python-requests/2.32.5" "3cc716d1-2040-4646-ba05-ac7cb0fce2cd" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.2:36712 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:20.658Z] "POST /maas-api/v1/api-keys HTTP/1.1" 401 - - "-" 37 0 26 - "100.64.0.3" "python-requests/2.32.5" "5bc97b93-dea1-4edc-b4a8-90fcadd05dbd" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.133.0.25:443 100.64.0.3:21162 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:20.702Z] "POST /maas-api/v1/api-keys HTTP/1.1" 401 - - "-" 35 0 3 - "100.64.0.4" "python-requests/2.32.5" "2241269e-8c40-427d-93b5-bcd73ca9663a" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.133.0.25:443 100.64.0.4:13454 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:20.722Z] "POST /maas-api/v1/api-keys HTTP/1.1" 401 - - "-" 36 0 2 - "100.64.0.3" "python-requests/2.32.5" "f006f57f-c8c8-4860-938a-6a70146a4d74" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.133.0.25:443 100.64.0.3:60240 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:21.067Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 28 312 8 - "100.64.0.2" "python-requests/2.32.5" "ede7ad74-93c1-4daa-a69a-804b06f813c6" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.2:36724 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T03:01:21.272335Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.272383Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.275270Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.275378Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.275564Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.279166Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.279222Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:21.279232Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.297291Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.297330Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.301767Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.301904Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.304036Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.304068Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.310396Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.310532Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.310634Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.312098Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.312153Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:21.312162Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:21.313384Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.313439Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:21.313450Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.330174Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.330208Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.336790Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.336948Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.337187Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.337218Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.338529Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.338555Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.338760Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.341626Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.341758Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.341961Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.342339Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.423757Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.423804Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.427219Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.427329Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.427522Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.431214Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.431274Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:21.431285Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.449679Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.449711Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.452693Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.452791Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.457519Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:21.457569Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:21.457579Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-25T03:01:21.271Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 313 8 - "100.64.0.4" "python-requests/2.32.5" "a7edd00a-8ac4-45d5-8bec-224cc056fd99" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.4:59002 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:21.303Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 8 - "10.132.0.53" "Go-http-client/1.1" "8e1cdb09-148a-4f37-ad82-72b2046d9b4c" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.30:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:47828 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-25T03:01:21.296Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 17 - "100.64.0.4" "python-requests/2.32.5" "c10b680f-088c-4937-b2aa-86badec90142" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.4:13456 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:21.329Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 123 319 12 - "100.64.0.2" "python-requests/2.32.5" "5ceb2c74-9078-4ed9-a04b-d73319bb0f5a" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.30:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:33058 10.133.0.25:443 100.64.0.2:46972 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-25T03:01:21.422Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 314 8 - "100.64.0.3" "python-requests/2.32.5" "3705c5cd-9834-49d0-971a-f527f89bcfbd" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.3:50312 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:21.448Z] "DELETE /v1/api-keys/837496b0-d2d7-4b00-a9b6-eaa94611f883 HTTP/1.1" 200 - via_upstream - "-" 0 251 8 - "100.64.0.3" "python-requests/2.32.5" "42424055-6959-49ee-b77f-13cbb237f66e" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.3:21176 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T03:01:24.477503Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.477554Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.481580Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.481663Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.483576Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.483631Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:24.483641Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.611120Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:24.611167Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:24.624324Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:24.624362Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:24.763181Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.763224Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.766100Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.766208Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.766394Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.770103Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.770156Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:24.770166Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.787252Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.787288Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.789955Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.790063Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.790227Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.793582Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.793633Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:24.793643Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.875275Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.875321Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.877991Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.878117Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.878305Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.881788Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.881844Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:24.881855Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.900371Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.900408Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.903422Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.903524Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.907409Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.907461Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:24.907471Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:24.924415Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:24.924454Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:24.927165Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:24.927270Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:24.929778Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:24.929828Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:24.929838Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.010346Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.010393Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.013364Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.013480Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.013674Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.017329Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.017386Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:25.017396Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.034521Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.034560Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.038226Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.038335Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.040422Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.040454Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.046362Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.046497Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.046594Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.048031Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.048085Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:25.048095Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.049278Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.049337Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:25.049348Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.126039Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.126091Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.129368Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.129482Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.129666Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.133139Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.133194Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:25.133204Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.153391Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.153426Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.156852Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.156990Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.158675Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.158724Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:25.158734Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.178858Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.178911Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.182101Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.182204Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.184150Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.184181Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.189946Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.190084Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.190175Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.191485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.191535Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:25.191546Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.192697Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.192748Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:25.192758Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-25T03:01:24.475Z] "GET /v1/models HTTP/1.1" 500 - via_upstream - "-" 0 96 7 - "100.64.0.4" "python-requests/2.32.5" "60b1126e-4845-42cf-95a3-c33cd4922a41" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.4:62568 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:24.609Z] "POST /maas-api/v1/api-keys HTTP/1.1" 401 - - "-" 38 0 14 - "100.64.0.2" "python-requests/2.32.5" "c412e9de-e4b8-4f4c-86e8-70e534bbab6b" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.133.0.25:443 100.64.0.2:8706 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:24.762Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 34 319 8 - "100.64.0.4" "python-requests/2.32.5" "bef47cbb-0c5f-4c4e-840f-0249e691db5d" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.4:15452 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:24.786Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 32 317 7 - "100.64.0.2" "python-requests/2.32.5" "2655f11a-5095-44cb-b45b-d5ec27006422" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.2:47218 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:24.874Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 31 316 7 - "100.64.0.3" "python-requests/2.32.5" "775bac00-9cd6-4b6a-9b59-410208956d23" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.3:50314 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:24.899Z] "DELETE /v1/api-keys/0f19d29e-b8f0-47f8-84ea-6f77161d4ef0 HTTP/1.1" 200 - via_upstream - "-" 0 253 8 - "100.64.0.3" "python-requests/2.32.5" "b14ced07-672e-456f-aca7-c4786caad5b8" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.3:23446 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:24.923Z] "DELETE /v1/api-keys/0f19d29e-b8f0-47f8-84ea-6f77161d4ef0 HTTP/1.1" 404 - via_upstream - "-" 0 29 6 - "100.64.0.2" "python-requests/2.32.5" "2e333eee-8c4b-4361-acfe-fb3b25cb60d9" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.2:8718 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:25.009Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 313 8 - "100.64.0.4" "python-requests/2.32.5" "02d6489b-1449-45a7-af58-9d5672e0cf28" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.4:15456 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:25.039Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 8 - "10.132.0.53" "Go-http-client/1.1" "1ce89b2e-a23a-4feb-a8b2-4f8a70660741" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.30:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:47828 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-25T03:01:25.033Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 15 - "100.64.0.3" "python-requests/2.32.5" "5d3e2f78-6e29-4af6-bca6-d47febbd39fc" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.3:50324 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:25.124Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 314 8 - "100.64.0.4" "python-requests/2.32.5" "c3ec23be-2d7e-409b-a5f7-893d7c102646" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.4:62572 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:25.152Z] "GET /v1/models HTTP/1.1" 403 - via_upstream - "-" 0 89 6 - "100.64.0.2" "python-requests/2.32.5" "082dc246-772b-40ef-8dea-824be043772e" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.2:8730 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:25.183Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 8 - "10.132.0.53" "Go-http-client/1.1" "6d2508a0-4143-4fed-8445-1f694636729e" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.30:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:47828 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-25T03:01:25.177Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 14 - "100.64.0.2" "python-requests/2.32.5" "c50a017f-bbf8-44a6-b5d8-d9b97b64274c" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.2:47222 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-25T03:01:25.312180Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.312225Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.315292Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.315444Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.315676Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.319164Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.319221Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-25T03:01:25.319231Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-25T03:01:25.338615Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.338664Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.342095Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.342212Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.344265Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.344298Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.350265Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.350407Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.350507Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.351949Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.352003Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:25.352014Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.353129Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.353179Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:25.353189Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.368731Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.368765Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.371818Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.371933Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.373951Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.373986Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.379434Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.379571Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.379668Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.381053Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.381105Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:25.381115Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.382244Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.382291Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:25.382302Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.464414Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.464464Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.467243Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.467359Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.467552Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.471024Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-25T03:01:25.471086Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-25T03:01:25.471098Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-25T03:01:25.311Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 313 8 - "100.64.0.3" "python-requests/2.32.5" "14ce004c-5ecf-4026-a98e-39bddc15e0fa" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:44022 10.133.0.25:443 100.64.0.3:23462 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:25.343Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 8 - "10.132.0.53" "Go-http-client/1.1" "85831447-220f-4a3a-a9e5-d96e41f99cc1" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.30:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:47828 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-25T03:01:25.337Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 15 - "100.64.0.4" "python-requests/2.32.5" "5c184d62-b4cd-41a4-9bb6-da88377cffdc" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.4:15472 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:25.373Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 7 - "10.132.0.53" "Go-http-client/1.1" "c7b30105-8d6e-4144-8cd9-b90a98f2819e" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.133.0.30:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.25:47828 10.133.0.25:443 10.132.0.53:46726 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-25T03:01:25.367Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 14 - "100.64.0.3" "python-requests/2.32.5" "db47471b-c7bd-4eae-bfbf-e425d34c9a77" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.3:50330 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-25T03:01:25.463Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 36 321 7 - "100.64.0.2" "python-requests/2.32.5" "817a4123-2b56-4aa2-81d2-2194ff5dc75f" "maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com" "10.132.0.53:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.25:58914 10.133.0.25:443 100.64.0.2:8740 maas.apps.afb40153-236a-4bab-9e0d-84bdea5ab30b.prod.konfluxeaas.com opendatahub.maas-api-route.1