[2026-05-20T20:54:53.485Z] "GET /v1/models HTTP/1.1" 403 - via_upstream - "-" 0 89 16 - "100.64.0.4" "python-requests/2.32.5" "ebecfdb1-37b2-458f-94ad-eede6903a8b6" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.4:29102 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:55:07.599286Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:07.599333Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:07.615127Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:07.615301Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:07.615549Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:07.620097Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:07.620160Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:07.620171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:55:07.598Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 45 329 22 - "100.64.0.2" "python-requests/2.32.5" "772142bd-ba4e-4fe8-bd05-850561c9cb5d" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.2:19370 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:55:15.647932Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:15.647982Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:15.662676Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:15.662815Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:15.673081Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:15.673123Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:15.680562Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:15.680700Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:15.680814Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:15.682336Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:15.682396Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:15.682407Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:15.683306Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:15.683367Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:55:15.683377Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-20T20:55:15.672Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 10 - "10.133.0.45" "Go-http-client/1.1" "76462ffb-c0a2-4905-b106-b233dc978787" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.132.0.48:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:53178 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-20T20:55:15.646Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 471 36 - "100.64.0.3" "python-requests/2.32.5" "1da2f9df-f0f4-96fc-b364-55bcceab3c64" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.3:26698 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:55:29.904998Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:29.905049Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:29.920912Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:29.921069Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:29.921340Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:29.925139Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:29.925218Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:55:29.925231Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:55:29.945313Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:29.945353Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:29.955821Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:29.955950Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:29.956168Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:29.959242Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:29.959301Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:29.959312Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:55:29.903Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 55 297 21 - "100.64.0.3" "python-requests/2.32.5" "ff2a679d-23cb-4431-9b03-79a2855235a5" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.3:30320 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:55:29.944Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 55 298 15 - "100.64.0.2" "python-requests/2.32.5" "873af120-c287-4137-af31-61bde99c101f" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.2:49458 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:55:37.989440Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:37.989494Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:37.993718Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:37.993853Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:37.995902Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:37.995940Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.003497Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.003634Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.003759Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.005270Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.005337Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:38.005349Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.006178Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.006257Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:38.006270Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.023238Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.023282Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.026856Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.026989Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.028896Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.028935Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.035825Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.035950Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.036059Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.037291Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.037349Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:38.037360Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.038161Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:38.038238Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:38.038252Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:55:37.994Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 10 - "10.133.0.45" "Go-http-client/1.1" "afb7a84a-6fa5-454c-8cdd-fd8b5cf238d8" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.132.0.48:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:53178 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-20T20:55:37.988Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 468 18 - "100.64.0.4" "python-requests/2.32.5" "9c8e7c28-0883-42a5-8c22-b34e2cc83a0e" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.4:47858 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:55:38.028Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 166 9 - "10.133.0.45" "Go-http-client/1.1" "60156d8a-6a9c-489d-8ba0-397f02912739" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.48:8000" outbound|8000||e2e-distinct-2-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:55812 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-distinct-2-simulated-kserve-route.2 [2026-05-20T20:55:38.022Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 345 16 - "100.64.0.2" "python-requests/2.32.5" "a0003e26-2d63-4de9-bc5b-65fff461c444" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.2:23706 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:55:54.613613Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.613669Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.629305Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.629454Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.631649Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.631685Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.641837Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.641981Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.643567Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.643641Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:54.643654Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.645028Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.645065Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.655013Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.655153Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.656418Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.656476Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:54.656487Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.657702Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.657736Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.667857Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.667999Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.670072Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.670141Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:54.670152Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.671066Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:55:54.671129Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:55:54.671141Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:55:54.630Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 12 - "10.133.0.45" "Go-http-client/1.1" "24023585-e1e3-4e2e-a665-cd558f9509c4" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.132.0.48:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:53178 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-20T20:55:54.644Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 166 12 - "10.133.0.45" "Go-http-client/1.1" "357a040d-5d20-453e-90c2-532766e2fe7c" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.48:8000" outbound|8000||e2e-distinct-2-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:55812 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-distinct-2-simulated-kserve-route.2 [2026-05-20T20:55:54.656Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 13 - "10.133.0.45" "Go-http-client/1.1" "7cdef0ac-cc4c-4c7f-9444-11e6b55da87a" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:52984 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-20T20:55:54.612Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 1361 58 - "100.64.0.3" "python-requests/2.32.5" "991359d2-2e4a-48ad-b2ca-6dab2ad87629" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.3:60092 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:55:58.161Z] "GET /portal/redlion HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "100.64.0.2" "Mozilla/5.0 zgrab/0.x" "b8c9aa5c-e7a6-4b6c-aa3d-cbac6eb8bc15" "3.83.196.253" "-" - - 10.133.0.35:80 100.64.0.2:1280 - - 2026-05-20T20:56:11.187519Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:11.187568Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:11.203070Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:11.203307Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:11.205677Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.205713Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.216282Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.216423Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.217918Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.217978Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:56:11.217989Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.218903Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:11.218958Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:56:11.218969Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:11.240054Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.240100Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.250523Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.250651Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.252618Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.252649Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.263540Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.263664Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.264974Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.265029Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:56:11.265039Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.265855Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:11.265900Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:56:11.265911Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:56:11.204Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 13 - "10.133.0.45" "Go-http-client/1.1" "033d87cc-37a5-4348-b363-4bdf74aa7342" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.132.0.48:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:53178 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-20T20:56:11.186Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 470 32 - "100.64.0.4" "python-requests/2.32.5" "a403064a-ed7c-4d3a-96e1-e35b45dd3914" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.4:19488 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:56:11.251Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 166 13 - "10.133.0.45" "Go-http-client/1.1" "4029ef1a-4f3d-97fc-a13d-b926ad73cf5a" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.48:8000" outbound|8000||e2e-distinct-2-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:55812 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-distinct-2-simulated-kserve-route.2 [2026-05-20T20:56:11.239Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 347 26 - "100.64.0.4" "python-requests/2.32.5" "d2c39a32-a4e2-4e2b-8f51-5a09aeebf464" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.4:24138 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:56:20.293124Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:20.293177Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:20.294770Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:20.294802Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:56:20.291Z] "GET /maas-api/v1/models HTTP/1.1" 401 - - "-" 0 0 2 - "100.64.0.3" "python-requests/2.32.5" "5560085b-813a-4ff8-9948-df04c494291e" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.133.0.35:443 100.64.0.3:26784 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:56:36.905128Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.905179Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.920647Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.920857Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.921185Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.950817Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.950888Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:56:36.950899Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.971900Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.971946Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.982135Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.982308Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.982634Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.982669Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.984416Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.984443Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.984712Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.986226Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.986370Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.986562Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:36.987179Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.002827Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.002870Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.009833Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.009958Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.010289Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.010322Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.010822Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.010847Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.030977Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:37.031020Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:37.037944Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:37.038074Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:37.038403Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:37.038438Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:37.038944Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:37.038971Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:37.060591Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.060636Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.064116Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.064245Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.066480Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.066513Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.073065Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.073186Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.073321Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.075481Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.075539Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:56:37.075549Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.076842Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:37.076893Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:56:37.076904Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:56:36.904Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 331 46 - "100.64.0.2" "python-requests/2.32.5" "07321ea8-5381-493c-ac87-5b69df1b4a06" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.2:9416 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:56:36.971Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 15 - "100.64.0.2" "python-requests/2.32.5" "db3cefdc-6f74-4482-95a5-9efbfa1148e4" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:33788 10.133.0.35:443 100.64.0.2:22574 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.0 [2026-05-20T20:56:37.002Z] "POST /v1/completions HTTP/1.1" 429 - - "-" 66 18 8 - "100.64.0.3" "python-requests/2.32.5" "c22b1964-8170-4afc-96da-4835df38f32c" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:33788 10.133.0.35:443 100.64.0.3:49434 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.0 [2026-05-20T20:56:37.030Z] "POST /v1/completions HTTP/1.1" 429 - - "-" 66 18 8 - "100.64.0.4" "python-requests/2.32.5" "7fcd2916-775e-44cc-a80f-2710aadb23ef" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.35:443 100.64.0.4:35494 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.0 [2026-05-20T20:56:37.065Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 9 - "10.133.0.45" "Go-http-client/1.1" "532e83f8-a502-4b3a-8cd6-c874172c66ea" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:40248 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.2 [2026-05-20T20:56:37.059Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 547 17 - "100.64.0.4" "python-requests/2.32.5" "3d0feff6-9286-47cf-9af3-df0634c44b34" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.4:27752 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:56:45.938570Z info ads ADS: new connection for node:6 2026-05-20T20:56:45.942979Z info cache read certificate from file resource=file-root:system 2026-05-20T20:56:52.665827Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:52.665885Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:52.681652Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:52.681807Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:52.682080Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:52.685617Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:52.685682Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:56:52.685690Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:53.021108Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:53.021154Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:53.022340Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:53.022377Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:53.039991Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:53.040032Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:53.040935Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:53.040967Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:56:53.059440Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:53.059493Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:53.066673Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:53.066809Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:53.067125Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:53.067161Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:53.069036Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:53.069064Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:53.069363Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:53.070241Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:56:53.070383Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:56:53.070395Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:56:52.664Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 79 322 21 - "100.64.0.2" "python-requests/2.32.5" "568aef58-e8b7-477a-8d93-bf6641407e08" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.2:15520 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:56:53.020Z] "POST /llm/e2e-external-model/v1/chat/completions HTTP/1.1" 401 - - "-" 83 0 2 - "100.64.0.3" "python-requests/2.32.5" "51ba883d-1267-4f41-aec6-6bee82d8435d" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "-" outbound|443||httpbin.org - 10.133.0.35:443 100.64.0.3:30668 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-external-model.0 [2026-05-20T20:56:53.038Z] "POST /llm/e2e-external-model/v1/chat/completions HTTP/1.1" 401 - - "-" 83 0 2 - "100.64.0.4" "python-requests/2.32.5" "66486f0a-bff7-4758-9538-83f48ac98b07" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "-" outbound|443||httpbin.org - 10.133.0.35:443 100.64.0.4:11506 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-external-model.0 [2026-05-20T20:56:53.057Z] "POST /llm/e2e-external-model/v1/chat/completions HTTP/1.1" 404 - - "-" 83 98 12 - "100.64.0.2" "python-requests/2.32.5" "daab8f6c-67b5-42a9-a38d-458bdb7cd4a4" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "-" outbound|443||httpbin.org - 10.133.0.35:443 100.64.0.2:15522 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.e2e-external-model.0 2026-05-20T20:57:30.707669Z info ads ADS: "" 6 terminated with stream closed 2026-05-20T20:57:30.753213Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.753261Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.756182Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.756316Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.756523Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.769049Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.769121Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:30.769132Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.909929Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.909975Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.938393Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.938448Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.960292Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.960336Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.961663Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.961694Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:30.983287Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:30.983336Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:30.984619Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:30.984651Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:31.641669Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:31.641732Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:31.644613Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:31.644783Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:31.645055Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:31.648601Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:31.648662Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:31.648673Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:57:30.751Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 314 17 - "100.64.0.3" "python-requests/2.32.5" "a0c6e67e-6145-439d-871c-537c0e0518a4" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.3:40522 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:30.908Z] "POST /maas-api/v1/api-keys HTTP/1.1" 401 - - "-" 37 0 29 - "100.64.0.4" "python-requests/2.32.5" "34f6a87f-3d34-4c97-a993-e9d52410aea8" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.133.0.35:443 100.64.0.4:47228 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:30.959Z] "POST /maas-api/v1/api-keys HTTP/1.1" 401 - - "-" 35 0 2 - "100.64.0.3" "python-requests/2.32.5" "2a46458b-7bed-4902-bb0e-3e2e64f63b6c" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.133.0.35:443 100.64.0.3:18232 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:30.982Z] "POST /maas-api/v1/api-keys HTTP/1.1" 401 - - "-" 36 0 2 - "100.64.0.2" "python-requests/2.32.5" "3337a9bb-ec1d-4b94-9166-c7c7484d6948" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.133.0.35:443 100.64.0.2:48122 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:31.639Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 28 312 8 - "100.64.0.3" "python-requests/2.32.5" "d76ecc00-862e-4d0d-bdfb-f0e20436ad12" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.3:40524 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:57:32.017709Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:32.017757Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:32.020526Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:32.020652Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:32.020867Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:32.024110Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:32.024171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:57:32.024182Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:32.043293Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.043346Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.047731Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.047875Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.049738Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.049780Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.056179Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.056362Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.056486Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.057742Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.057813Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:32.057829Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.058781Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.058846Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:32.058862Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.075310Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.075366Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.082211Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.082381Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.082714Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.082758Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.084132Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.084163Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.084442Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.086166Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.086331Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.086528Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.087045Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.241500Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.241547Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.244043Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.244171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.244394Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.247607Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.247684Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:32.247701Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.267339Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.267390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.269808Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.269924Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.274276Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:32.274339Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:32.274349Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:57:32.016Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 314 7 - "100.64.0.2" "python-requests/2.32.5" "c695443a-6e2b-4c11-9af5-44da13ad8520" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.2:24132 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:32.048Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 8 - "10.133.0.45" "Go-http-client/1.1" "7f61399a-f67c-4c0d-af13-93a7b5249d20" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:52984 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-20T20:57:32.042Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 16 - "100.64.0.4" "python-requests/2.32.5" "2f6b199f-5d86-4eb4-ba7b-0d47be639092" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.4:47238 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:32.074Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 123 332 12 - "100.64.0.2" "python-requests/2.32.5" "9535993b-0ba8-4468-bd90-f3472c8133b2" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:52984 10.133.0.35:443 100.64.0.2:48128 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-20T20:57:32.240Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 313 7 - "100.64.0.3" "python-requests/2.32.5" "a0a6bbb8-a406-4ea8-a4f2-3d13519a759f" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.3:40536 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:32.266Z] "DELETE /v1/api-keys/34d84b24-ce4c-4b41-a4f3-c59c589364b9 HTTP/1.1" 200 - via_upstream - "-" 0 251 8 - "100.64.0.4" "python-requests/2.32.5" "040f6db8-60ec-478b-87a2-6faf0197680f" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.4:49894 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:57:35.293326Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.293377Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.297636Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.297727Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.299173Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.299263Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:57:35.299275Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.489857Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.489908Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.505587Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.505633Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:57:35.292Z] "GET /v1/models HTTP/1.1" 500 - via_upstream - "-" 0 96 7 - "100.64.0.3" "python-requests/2.32.5" "935c31e3-c099-401f-bd0e-d8c0d7ade5e7" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.3:36806 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:35.488Z] "POST /maas-api/v1/api-keys HTTP/1.1" 401 - - "-" 38 0 17 - "100.64.0.4" "python-requests/2.32.5" "c667fdf8-35b4-4f68-a4e3-d2bcb772088f" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.133.0.35:443 100.64.0.4:42762 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:57:35.776046Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.776105Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.779042Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.779172Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.779409Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.782635Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.782698Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:57:35.782709Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.808789Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.808840Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.811354Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.811489Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.811715Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.814979Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.815043Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:57:35.815054Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:35.959571Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.959618Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.962431Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.962563Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.962772Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.965865Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.965925Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:35.965936Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.981542Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.981587Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.984050Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.984177Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.987772Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:35.987837Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:35.987847Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.005396Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.005445Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.007801Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.007923Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.009932Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.009989Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:57:36.009999Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.159326Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.159374Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.161820Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.161937Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.162132Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.165164Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.165253Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:36.165265Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.185317Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.185362Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.188881Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.189005Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.190890Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.190925Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.197627Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.197773Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.197891Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.199187Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.199262Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:36.199272Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.200148Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.200223Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:57:36.200235Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.341301Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.341353Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.343921Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.344038Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.344269Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.347297Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.347358Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:36.347369Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.362931Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.362975Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.366588Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.366716Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.368577Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.368638Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:36.368649Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.386899Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.386943Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.390444Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.390586Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.393238Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.393283Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.400559Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.400710Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.400830Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.402182Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.402274Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:36.402285Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.403284Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.403337Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:36.403345Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.545046Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.545092Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.547825Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.547944Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.548160Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.551462Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.551527Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:57:36.551537Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.566766Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.566812Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.570567Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.570721Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.572673Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.572707Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.578967Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.579113Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.579241Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.580479Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.580539Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:36.580549Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.581517Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.581575Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:36.581586Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.596360Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.596404Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.599621Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.599745Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.601814Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.601852Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.621711Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.621912Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.622066Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.623831Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.623917Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:36.623935Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.625135Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-20T20:57:36.625221Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-20T20:57:36.625233Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-20T20:57:35.774Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 34 319 7 - "100.64.0.2" "python-requests/2.32.5" "c509c953-33c2-40d1-b1a1-de337fd95601" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.2:13252 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:35.807Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 32 317 7 - "100.64.0.2" "python-requests/2.32.5" "f1dfba30-b378-414a-8464-3ffd14bca407" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.2:61078 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:35.958Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 31 315 7 - "100.64.0.3" "python-requests/2.32.5" "0a160428-f90b-434c-93e6-db5386d5f5f8" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.3:37492 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:35.980Z] "DELETE /v1/api-keys/c270fde5-4ffa-4a53-bbcc-7a6b1da37c5f HTTP/1.1" 200 - via_upstream - "-" 0 253 7 - "100.64.0.4" "python-requests/2.32.5" "b1d243b9-d081-4e5e-ab78-d6e681c84e0a" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.4:42776 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:36.004Z] "DELETE /v1/api-keys/c270fde5-4ffa-4a53-bbcc-7a6b1da37c5f HTTP/1.1" 404 - via_upstream - "-" 0 29 5 - "100.64.0.2" "python-requests/2.32.5" "1297a4fe-790c-4967-bad2-2fc88d4cb6aa" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.2:61084 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:36.158Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 313 7 - "100.64.0.3" "python-requests/2.32.5" "3198bcea-91df-4d61-872b-933ca61135df" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.3:37496 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:36.190Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 9 - "10.133.0.45" "Go-http-client/1.1" "c0d0fc77-66ba-440d-95ab-4b8242984dbc" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:52984 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-20T20:57:36.184Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 16 - "100.64.0.4" "python-requests/2.32.5" "f02553e2-9e49-452e-bf93-ce6d888a6aec" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.4:42790 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:36.340Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 313 7 - "100.64.0.2" "python-requests/2.32.5" "cf949641-08f3-4105-9065-257d612f6a31" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.2:61086 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:36.361Z] "GET /v1/models HTTP/1.1" 403 - via_upstream - "-" 0 89 6 - "100.64.0.3" "python-requests/2.32.5" "c082826e-5f5d-47f2-a19f-4c92221c3f1c" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.3:37504 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:36.391Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 10 - "10.133.0.45" "Go-http-client/1.1" "5addf26a-baa8-43d7-a093-9afaf4caaa17" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:52984 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-20T20:57:36.385Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 17 - "100.64.0.4" "python-requests/2.32.5" "b21a8247-12d7-436e-b4a5-bf3de4a58a62" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.4:42796 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:36.543Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 29 313 7 - "100.64.0.2" "python-requests/2.32.5" "cce51330-b759-4a46-9f7e-2a281dd06132" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.2:61090 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:36.571Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 8 - "10.133.0.45" "Go-http-client/1.1" "0c6d1fe3-e425-470d-980d-8791366fe822" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:52984 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-20T20:57:36.565Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 15 - "100.64.0.3" "python-requests/2.32.5" "02b167d1-9b2c-43f4-96d9-d16296eed6e2" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.3:37512 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-20T20:57:36.600Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 23 - "10.133.0.45" "Go-http-client/1.1" "2190b484-2ab4-4cf4-8b63-82974d681bcc" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.46:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.35:52984 10.133.0.35:443 10.133.0.45:48346 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-20T20:57:36.595Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 29 - "100.64.0.4" "python-requests/2.32.5" "1950dc1b-10f7-472b-8f49-4c6140ff3090" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:44988 10.133.0.35:443 100.64.0.4:42800 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-20T20:57:36.768318Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.768373Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.770877Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.771017Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.771259Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.774462Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-20T20:57:36.774533Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-20T20:57:36.774548Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-20T20:57:36.767Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 36 319 7 - "100.64.0.2" "python-requests/2.32.5" "2b08779d-9a97-4ff9-b3b6-1e14102db9da" "maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com" "10.133.0.45:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.133.0.35:53440 10.133.0.35:443 100.64.0.2:61100 maas.apps.97ea4eb0-0d24-4be0-8c7d-505b7b528d33.prod.konfluxeaas.com opendatahub.maas-api-route.1