2026-05-21T02:17:31.547166Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:31.891729Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:31.891778Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:31.896401Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:31.896522Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:31.896731Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:31.900370Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:31.900428Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:17:31.900438Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:17:31.536Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 9 - "10.133.0.30" "Go-http-client/1.1" "dc29ecd6-e795-4566-89c2-ac7999528862" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:32974 10.132.0.24:443 10.133.0.30:59438 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-21T02:17:31.529Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 17 - "100.64.0.4" "python-requests/2.32.5" "8a62d9aa-5ae7-47c7-a5b3-be5e61b8b68a" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:59722 10.132.0.24:443 100.64.0.4:10600 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-21T02:17:31.890Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 33 318 9 - "100.64.0.3" "python-requests/2.32.5" "0c77fd62-4326-4a86-8c59-f71f6e331272" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:59722 10.132.0.24:443 100.64.0.3:36310 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:17:39.932901Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.932954Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.937220Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.937364Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.939550Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.939585Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.946546Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.946691Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.946790Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.948409Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.948468Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:17:39.948478Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.949589Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:39.949643Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:17:39.949653Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:17:39.938Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 9 - "10.133.0.30" "Go-http-client/1.1" "c08f53e8-b68d-4f07-a955-5703b9a17ab7" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:32974 10.132.0.24:443 10.133.0.30:59438 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-21T02:17:39.931Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 590 17 - "100.64.0.3" "python-requests/2.32.5" "3893aca1-60bd-4b9c-a43e-4b4a62bf7f3a" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:59722 10.132.0.24:443 100.64.0.3:64378 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:17:40.625431Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:40.625477Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:40.630313Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:40.630431Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:40.630630Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:40.634352Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:40.634406Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:17:40.634416Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:17:40.624Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 79 321 10 - "100.64.0.2" "python-requests/2.32.5" "ee15c272-43cf-4ba4-9076-5c4fd4e9932d" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:59722 10.132.0.24:443 100.64.0.2:30816 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:17:48.673183Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.673234Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.678020Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.678136Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.680323Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.680353Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.680366Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.687682Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.687803Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.687904Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.687979Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.689606Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.689630Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.689675Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:17:48.689684Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.689697Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:17:48.689705Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.690849Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:48.690901Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:17:48.690912Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:17:48.679Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 10 - "10.133.0.30" "Go-http-client/1.1" "d69042fd-0075-4c24-9a6b-27990ebadd85" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:32974 10.132.0.24:443 10.133.0.30:59438 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-21T02:17:48.672Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 474 18 - "100.64.0.4" "python-requests/2.32.5" "ca7a6a56-5a88-4c9e-8487-efe8d39c4515" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:59722 10.132.0.24:443 100.64.0.4:24780 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:17:57.800547Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:57.800592Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:57.805369Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:57.805485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:57.805689Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:57.810007Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:17:57.810062Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:17:57.810072Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:17:57.799Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 85 327 10 - "100.64.0.3" "python-requests/2.32.5" "f287dce1-71b1-461e-9398-ad4ae6965fb5" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:59722 10.132.0.24:443 100.64.0.3:47660 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:18:14.065166Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:18:14.065216Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:18:14.069787Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:18:14.069900Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:18:14.071836Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:18:14.071887Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:18:14.071897Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:18:14.064Z] "GET /v1/models HTTP/1.1" 403 - via_upstream - "-" 0 82 7 - "100.64.0.2" "python-requests/2.32.5" "51d3ac50-365a-413a-9e68-7648ff85f066" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:59722 10.132.0.24:443 100.64.0.2:53148 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:18:31.152999Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:18:31.153048Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:18:31.157804Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:18:31.157934Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:18:31.161932Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:18:31.161994Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:18:31.162006Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-21T02:18:31.151Z] "GET /v1/models HTTP/1.1" 403 - via_upstream - "-" 0 89 10 - "100.64.0.4" "python-requests/2.32.5" "5a22a821-02fc-472a-8919-fa9e8fac8df1" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.4:33062 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:18:48.284534Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:18:48.284579Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:18:48.288910Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:18:48.289029Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:18:48.291139Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:18:48.291192Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:18:48.291202Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:18:48.283Z] "GET /v1/models HTTP/1.1" 403 - via_upstream - "-" 0 82 7 - "100.64.0.2" "python-requests/2.32.5" "755e75d7-8bcf-4eb7-ab99-8261860ad9d6" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:59722 10.132.0.24:443 100.64.0.2:39568 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:19:05.611339Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:05.611387Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:05.616576Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:05.616717Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:05.618861Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:05.618933Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:19:05.618949Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-21T02:19:05.610Z] "GET /v1/models HTTP/1.1" 403 - via_upstream - "-" 0 89 8 - "100.64.0.4" "python-requests/2.32.5" "630bc1a3-e0c0-424c-8f86-b16e1bf35b45" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.4:1716 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:19:19.603757Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:19.603810Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:19.608485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:19.608607Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:19.608817Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:19.612858Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:19.612918Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:19:19.612929Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-21T02:19:19.602Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 45 330 10 - "100.64.0.3" "python-requests/2.32.5" "a421ea0b-94e6-47f1-ad5b-3a5bf5117058" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.3:41830 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:19:27.646282Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:27.646339Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:27.660717Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:27.660867Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:27.672055Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:27.672093Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:27.678959Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:27.679090Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:27.679198Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:27.680884Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:27.680939Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:19:27.680949Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:27.682133Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:27.682201Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:19:27.682217Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-21T02:19:27.671Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 9 - "10.133.0.30" "Go-http-client/1.1" "5af7339d-e40f-99d9-972d-df1ab22ec1c6" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.32:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:58994 10.132.0.24:443 10.133.0.30:49750 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-21T02:19:27.645Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 471 37 - "100.64.0.3" "python-requests/2.32.5" "9d5844a9-f5fc-406c-a4d3-bb9acf694a2d" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.3:12418 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:19:39.644413Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:39.644462Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:39.650091Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:39.650215Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:39.650449Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:39.655365Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:39.655427Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:19:39.655437Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:39.677989Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:39.678028Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:39.681843Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:39.681960Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:39.682130Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:39.685955Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:39.686004Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:19:39.686014Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:19:39.643Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 55 297 12 - "100.64.0.2" "python-requests/2.32.5" "6bc8efe4-3b5e-41fb-9a9a-36e58141ee9e" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.2:12742 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-21T02:19:39.677Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 55 298 9 - "100.64.0.2" "python-requests/2.32.5" "0828fee1-d1bd-4635-a6cd-66ca3fa84027" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:59722 10.132.0.24:443 100.64.0.2:17830 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:19:47.717934Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:47.717987Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:47.722470Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:47.722595Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:47.725014Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.725055Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.731626Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.731748Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.731854Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.733495Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.733556Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:19:47.733567Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.734813Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:47.734874Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:19:47.734885Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:19:47.754683Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.754716Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.757905Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.757998Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.759938Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.759970Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.766371Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.766484Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.766576Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.768893Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.768940Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:19:47.768950Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.769948Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:19:47.769995Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:19:47.770006Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:19:47.724Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 9 - "10.133.0.30" "Go-http-client/1.1" "a869efe3-9584-4e9e-8bcc-e19f3d1af046" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.32:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:58994 10.132.0.24:443 10.133.0.30:49750 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-21T02:19:47.716Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 468 18 - "100.64.0.4" "python-requests/2.32.5" "cf7b8118-aa44-4aaf-b3ec-1762770121d4" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.4:53676 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-21T02:19:47.759Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 166 9 - "10.133.0.30" "Go-http-client/1.1" "2c7aa0c8-b756-464f-a17f-676c768d8e4d" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.31:8000" outbound|8000||e2e-distinct-2-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:41804 10.132.0.24:443 10.133.0.30:49750 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-distinct-2-simulated-kserve-route.2 [2026-05-21T02:19:47.753Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 345 16 - "100.64.0.3" "python-requests/2.32.5" "74cec543-ab9d-462f-9685-d990cf5db57f" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:59722 10.132.0.24:443 100.64.0.3:61544 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:20:04.155461Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:04.155509Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:04.160676Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:04.160792Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:04.163234Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.163296Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.171130Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.171270Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.172850Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.172905Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:20:04.172915Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.174281Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.174309Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.180960Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.181064Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.182485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.182533Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:20:04.182543Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.183758Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.183788Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.191414Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.191517Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.192884Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.192929Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:20:04.192939Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:04.194014Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:04.194076Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:20:04.194089Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-21T02:20:04.162Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 10 - "10.133.0.30" "Go-http-client/1.1" "9ab3637b-880c-445e-9bd3-d8b5f77ab60f" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.32:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:58994 10.132.0.24:443 10.133.0.30:49750 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-21T02:20:04.173Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 166 9 - "10.133.0.30" "Go-http-client/1.1" "0134e5d2-362a-4e16-b88b-dcc4bacae889" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.31:8000" outbound|8000||e2e-distinct-2-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:41804 10.132.0.24:443 10.133.0.30:49750 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-distinct-2-simulated-kserve-route.2 [2026-05-21T02:20:04.183Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 9 - "10.133.0.30" "Go-http-client/1.1" "771739f8-db46-47a4-88e6-d87d7164dad2" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:32974 10.132.0.24:443 10.133.0.30:49750 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-21T02:20:04.154Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 1361 39 - "100.64.0.4" "python-requests/2.32.5" "a7b19eaf-ee93-4bb8-97cc-33966a74e38a" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.4:59314 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:20:20.567515Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.567570Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.572785Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.572921Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.575580Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.575622Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.583464Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.583603Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.585357Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.585427Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:20:20.585439Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.586717Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.586775Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:20:20.586786Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.608464Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.608498Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.612660Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.612765Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.614829Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.614863Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.622470Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.622581Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.624134Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.624184Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:20:20.624194Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:20.625373Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:20.625425Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:20:20.625435Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-21T02:20:20.574Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 162 10 - "10.133.0.30" "Go-http-client/1.1" "e411c07d-8094-47c3-872e-0a2e3d11b2de" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.32:8000" outbound|8000||e2e-distinct-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:58994 10.132.0.24:443 10.133.0.30:49750 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-distinct-simulated-kserve-route.2 [2026-05-21T02:20:20.566Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 470 20 - "100.64.0.2" "python-requests/2.32.5" "95ff1d02-78f1-4e36-9321-a5d3c518d93e" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.2:20604 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-21T02:20:20.614Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 166 10 - "10.133.0.30" "Go-http-client/1.1" "d576534e-a6a7-4474-ad95-2a92934e9b1e" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.31:8000" outbound|8000||e2e-distinct-2-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:41804 10.132.0.24:443 10.133.0.30:49750 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-distinct-2-simulated-kserve-route.2 [2026-05-21T02:20:20.607Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 347 17 - "100.64.0.4" "python-requests/2.32.5" "9ab2d079-2e13-9c5d-b475-133f2893acdd" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.4:26240 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-21T02:20:27.776Z] "GET / HTTP/1.1" 404 NR route_not_found - "-" 0 0 1 - "100.64.0.3" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1" "c1f07c92-4026-4148-9f11-c4487730b0bf" "52.206.177.128" "-" - - 10.132.0.24:80 100.64.0.3:49436 - - 2026-05-21T02:20:29.633598Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:29.633652Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:29.634969Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:29.635008Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-21T02:20:29.632Z] "GET /maas-api/v1/models HTTP/1.1" 401 - - "-" 0 0 2 - "100.64.0.3" "python-requests/2.32.5" "7fb73c87-3c56-4b61-a9a6-67126dd2bf1b" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "-" outbound|8443||maas-api.opendatahub.svc.cluster.local - 10.132.0.24:443 100.64.0.3:25600 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:20:46.228226Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.228304Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.234788Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.234929Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.235154Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.240336Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.240397Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:20:46.240408Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.263348Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.263384Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.270979Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.271100Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.271395Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.271427Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.272511Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.272541Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.272763Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.277326Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.277466Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.277654Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.278157Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.298083Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.298125Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.304414Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.304538Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.304783Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.304819Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.305153Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.305181Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.325520Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.325556Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.331621Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.331732Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.331955Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.331983Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.332238Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.332274Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.352008Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.352041Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.354915Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.355009Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.357565Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.357597Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.363367Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.363472Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.363561Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.365171Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.365219Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:20:46.365230Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:20:46.366662Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:20:46.366712Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:20:46.366722Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-21T02:20:46.226Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 88 330 13 - "100.64.0.3" "python-requests/2.32.5" "a456614e-b0cd-4cc3-b003-6a7a6956cbb7" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.3:18988 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-21T02:20:46.262Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 299 15 - "100.64.0.2" "python-requests/2.32.5" "8de7d7bc-5580-440a-8dae-9645603fe29b" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.134.0.62:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:42462 10.132.0.24:443 100.64.0.2:44440 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.0 [2026-05-21T02:20:46.297Z] "POST /v1/completions HTTP/1.1" 429 - - "-" 66 18 7 - "100.64.0.2" "python-requests/2.32.5" "139a96c6-82de-4c10-98a6-6bd7741d42d7" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.134.0.62:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:41276 10.132.0.24:443 100.64.0.2:25612 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.0 [2026-05-21T02:20:46.325Z] "POST /v1/completions HTTP/1.1" 429 - - "-" 66 18 7 - "100.64.0.4" "python-requests/2.32.5" "99118c79-9edb-4646-a1fe-85709b01677f" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.134.0.62:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.24:443 100.64.0.4:46156 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.0 [2026-05-21T02:20:46.357Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 8 - "10.133.0.30" "Go-http-client/1.1" "6eea4284-4f96-4a6f-84cb-0f2c0286abd3" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.134.0.62:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:42464 10.132.0.24:443 10.133.0.30:49750 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-unconfigured-facebook-opt-125m-simulated-kserve-route.2 [2026-05-21T02:20:46.351Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 547 15 - "100.64.0.4" "python-requests/2.32.5" "52ad3b8d-97af-4b35-83ef-082a211b58c8" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.4:40102 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:20:55.195623Z info ads ADS: new connection for node:6 2026-05-21T02:20:55.197991Z info cache read certificate from file resource=file-root:system 2026-05-21T02:21:01.928538Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:01.928583Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:01.933019Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:01.933134Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:01.933360Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:01.939070Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:01.939124Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:21:01.939135Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:21:01.927Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 79 321 11 - "100.64.0.3" "python-requests/2.32.5" "dfe6a7b1-6cbb-4893-881a-9f6912c5c1e9" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:57494 10.132.0.24:443 100.64.0.3:19680 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:21:02.287318Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:02.287363Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:02.288314Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:02.288345Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:02.310667Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.310703Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.311504Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.311539Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.331160Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.331194Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.337658Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.337783Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.338017Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.338048Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.339081Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.339113Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.339363Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.340450Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:02.340565Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:21:02.340575Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-21T02:21:02.286Z] "POST /llm/e2e-external-model/v1/chat/completions HTTP/1.1" 401 - - "-" 83 0 2 - "100.64.0.3" "python-requests/2.32.5" "96a8391c-d761-4034-935d-c7f504e33c11" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "-" outbound|443||httpbin.org - 10.132.0.24:443 100.64.0.3:21568 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-external-model.0 [2026-05-21T02:21:02.309Z] "POST /llm/e2e-external-model/v1/chat/completions HTTP/1.1" 401 - - "-" 83 0 1 - "100.64.0.2" "python-requests/2.32.5" "6c95fa58-f23c-4f43-99eb-a475af5df1bf" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "-" outbound|443||httpbin.org - 10.132.0.24:443 100.64.0.2:19870 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-external-model.0 [2026-05-21T02:21:02.330Z] "POST /llm/e2e-external-model/v1/chat/completions HTTP/1.1" 404 - - "-" 83 98 10 - "100.64.0.2" "python-requests/2.32.5" "1a1e5397-2a6d-4a06-b4f1-ecc52224c9d7" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "-" outbound|443||httpbin.org - 10.132.0.24:443 100.64.0.2:48064 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.e2e-external-model.0 2026-05-21T02:21:39.931477Z info ads ADS: "" 6 terminated with stream closed 2026-05-21T02:21:42.072466Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.072516Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.077516Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.077639Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.077853Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.092045Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.092118Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:21:42.092129Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.133386Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.133425Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.138060Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.138163Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.140205Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.140240Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.146804Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.146939Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.147030Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.148926Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.148974Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=18 2026-05-21T02:21:42.148984Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.150143Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.150210Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:21:42.150222Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.197032Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.197065Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 [2026-05-21T02:21:42.071Z] "POST /v1/api-keys HTTP/2" 201 - via_upstream - "-" 55 321 20 - "100.64.0.4" "curl/7.76.1" "14298073-b42b-4fa6-a8ae-684e6a8b6abe" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:57494 10.132.0.24:443 100.64.0.4:48594 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 [2026-05-21T02:21:42.139Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 150 9 - "10.133.0.30" "Go-http-client/1.1" "ae98ff6c-77af-49e8-8bfe-4e7cc8f29cc4" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:32974 10.132.0.24:443 10.133.0.30:49750 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.2 [2026-05-21T02:21:42.132Z] "GET /v1/models HTTP/2" 200 - via_upstream - "-" 0 590 17 - "100.64.0.4" "curl/7.76.1" "0ef9da3f-7c71-4c40-b5ab-78cc8bc54f75" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.4:10494 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1 2026-05-21T02:21:42.203658Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.203789Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.204013Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.204042Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.205074Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.205101Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.205312Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.207352Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.207487Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.207658Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.207963Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.244690Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.244729Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.252430Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.252604Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.252878Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.252912Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.253204Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.253228Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.253451Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.257666Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.257850Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.258088Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.258462Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.281065Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.281099Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.287591Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.287723Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.287952Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.287981Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.288295Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.288321Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.288514Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.290535Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.290674Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.290851Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.291134Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.318549Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.318580Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.325518Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.325646Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.325872Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.325899Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.326159Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.326183Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.326386Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.328146Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.328308Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.328482Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.328758Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.365211Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.365243Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.371923Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.372052Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.372293Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.372321Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.372564Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.372586Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.396595Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.396626Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.403240Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.403386Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.403601Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.403628Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.403886Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.403911Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.429606Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.429641Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.435860Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.435987Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.436209Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.436237Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.436507Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.436530Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=18 2026-05-21T02:21:42.460880Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.460913Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.467810Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.467939Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.468158Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.468186Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.468485Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.468509Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.494980Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.495013Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.501671Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.501809Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.502039Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.502068Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.502379Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.502403Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.526656Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.526688Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.532980Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.533105Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.533342Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.533370Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.533647Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.533670Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.564587Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.564618Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.571091Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.571227Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.571454Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.571483Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.571752Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.571777Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.599545Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.599576Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.600228Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.600266Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.625835Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.625866Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.629924Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.630015Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.635344Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 2026-05-21T02:21:42.635401Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Missing json property: /usage/total_tokens thread=17 2026-05-21T02:21:42.635412Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1140 wasm log openshift-ingress.kuadrant-maas-default-gateway: Span buffer full (100), dropping oldest span thread=17 [2026-05-21T02:21:42.196Z] "POST /v1/chat/completions HTTP/2" 200 - via_upstream - "-" 100 519 11 - "100.64.0.3" "curl/7.76.1" "c090f2c3-b192-406e-bdc0-4f2c492f6016" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:32974 10.132.0.24:443 100.64.0.3:17874 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.244Z] "POST /v1/chat/completions HTTP/2" 200 - via_upstream - "-" 100 347 14 - "100.64.0.3" "curl/7.76.1" "340763d2-795f-4899-b330-af308d566efc" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:32974 10.132.0.24:443 100.64.0.3:5136 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.280Z] "POST /v1/chat/completions HTTP/2" 200 - via_upstream - "-" 100 429 10 - "100.64.0.2" "curl/7.76.1" "3c42f188-0644-48da-9b47-57ee21bfa555" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:37264 10.132.0.24:443 100.64.0.2:43742 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.318Z] "POST /v1/chat/completions HTTP/2" 200 - via_upstream - "-" 100 345 10 - "100.64.0.4" "curl/7.76.1" "9e030e5a-029f-486f-bac1-83ef5ff2f6f3" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:37264 10.132.0.24:443 100.64.0.4:10500 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.364Z] "POST /v1/chat/completions HTTP/2" 429 - - "-" 100 18 7 - "100.64.0.2" "curl/7.76.1" "1d43a24f-06e2-41f9-8833-ab37d76dcd77" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:32974 10.132.0.24:443 100.64.0.2:7050 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.396Z] "POST /v1/chat/completions HTTP/2" 429 - - "-" 100 18 7 - "100.64.0.3" "curl/7.76.1" "db87d78b-62df-42e7-a99c-c64d76b36663" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:37264 10.132.0.24:443 100.64.0.3:5148 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.429Z] "POST /v1/chat/completions HTTP/2" 429 - - "-" 100 18 7 - "100.64.0.2" "curl/7.76.1" "e65d1be6-2979-4ce0-923e-1bebfd77922d" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.24:443 100.64.0.2:43756 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.460Z] "POST /v1/chat/completions HTTP/2" 429 - - "-" 100 18 8 - "100.64.0.4" "curl/7.76.1" "41afe3d9-ba45-46a3-96bd-7103b10741d5" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.24:443 100.64.0.4:48606 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.494Z] "POST /v1/chat/completions HTTP/2" 429 - - "-" 100 18 8 - "100.64.0.4" "curl/7.76.1" "1b3159ff-0e70-406f-8613-f51044e687b1" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:37200 10.132.0.24:443 100.64.0.4:10502 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.526Z] "POST /v1/chat/completions HTTP/2" 429 - - "-" 100 18 7 - "100.64.0.3" "curl/7.76.1" "dabf3dca-9a7e-43ac-b745-d37fe550dfa6" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.24:443 100.64.0.3:17880 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.564Z] "POST /v1/chat/completions HTTP/2" 429 - - "-" 100 18 7 - "100.64.0.2" "curl/7.76.1" "0c5b8dc0-230e-44e0-93e1-a62efe1ad6fe" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.29:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.24:37206 10.132.0.24:443 100.64.0.2:7058 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.599Z] "POST /llm/facebook-opt-125m-simulated/v1/chat/completions HTTP/2" 401 - - "-" 100 0 1 - "100.64.0.4" "curl/7.76.1" "3d7214a1-b84e-42a6-aa3f-45287f6e7b81" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.24:443 100.64.0.4:48610 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com llm.facebook-opt-125m-simulated-kserve-route.1 [2026-05-21T02:21:42.624Z] "DELETE /v1/api-keys/059e2536-d4ef-4c49-8c92-6aefcd618123 HTTP/2" 200 - via_upstream - "-" 0 334 10 - "100.64.0.3" "curl/7.76.1" "d481a4a7-87cb-4764-8341-08cc52af6896" "maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com" "10.133.0.30:8443" outbound|8443||maas-api.opendatahub.svc.cluster.local 10.132.0.24:54584 10.132.0.24:443 100.64.0.3:5158 maas.apps.66986e69-0d2c-4949-9a6e-40b3d01e4960.prod.konfluxeaas.com opendatahub.maas-api-route.1