2026-11-04T01:58:28.409829Z info FLAG: --concurrency="0" 2026-11-04T01:58:28.409846Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-04T01:58:28.409850Z info FLAG: --help="false" 2026-11-04T01:58:28.409852Z info FLAG: --log_as_json="false" 2026-11-04T01:58:28.409855Z info FLAG: --log_caller="" 2026-11-04T01:58:28.409857Z info FLAG: --log_output_level="default:info" 2026-11-04T01:58:28.409859Z info FLAG: --log_stacktrace_level="default:none" 2026-11-04T01:58:28.409863Z info FLAG: --log_target="[stdout]" 2026-11-04T01:58:28.409865Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-04T01:58:28.409868Z info FLAG: --outlierLogPath="" 2026-11-04T01:58:28.409870Z info FLAG: --profiling="true" 2026-11-04T01:58:28.409872Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-04T01:58:28.409874Z info FLAG: --proxyLogLevel="warning" 2026-11-04T01:58:28.409876Z info FLAG: --serviceCluster="istio-proxy" 2026-11-04T01:58:28.409879Z info FLAG: --stsPort="0" 2026-11-04T01:58:28.409880Z info FLAG: --templateFile="" 2026-11-04T01:58:28.409882Z info FLAG: --tokenManagerPlugin="" 2026-11-04T01:58:28.409885Z info FLAG: --vklog="0" 2026-11-04T01:58:28.409887Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-04T01:58:28.410022Z info Proxy role ips=[10.133.0.177] type=router id=maas-default-gateway-openshift-default-fc8c56547-dz9xw.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-04T01:58:28.410064Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-04T01:58:28.411140Z info cpu limit detected as 2, setting concurrency 2026-11-04T01:58:28.411328Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-04T01:58:28.411339Z info JWT policy is third-party-jwt 2026-11-04T01:58:28.411341Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-04T01:58:28.412512Z info Starting default Istio SDS Server 2026-11-04T01:58:28.412534Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-04T01:58:28.412551Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-04T01:58:28.413179Z info Opening status port 15020 2026-11-04T01:58:28.413377Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-04T01:58:28.413838Z info sds Starting SDS grpc server 2026-11-04T01:58:28.413858Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-04T01:58:28.414975Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-04T01:58:28.416955Z info Starting proxy agent 2026-11-04T01:58:28.418026Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-04T01:58:28.480835Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-04T01:58:28.510550Z info cache generated new workload certificate resourceName=default latency=96.605836ms ttl=23h59m59.48945418s 2026-11-04T01:58:28.510593Z info cache Root cert has changed, start rotating root cert 2026-11-04T01:58:28.510629Z info cache returned workload trust anchor from cache ttl=23h59m59.489371863s 2026-11-04T01:58:28.624478Z info ads ADS: new connection for node:1 2026-11-04T01:58:28.624537Z info cache returned workload certificate from cache ttl=23h59m59.375465184s 2026-11-04T01:58:28.624880Z info ads ADS: new connection for node:2 2026-11-04T01:58:28.624959Z info cache returned workload trust anchor from cache ttl=23h59m59.375042019s 2026-11-04T01:58:28.706369Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:58:36.925015Z info Readiness succeeded in 8.517714973s 2026-11-04T01:58:36.925311Z info Envoy proxy is ready 2026-11-04T01:58:37.867370Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:58:55.093533Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:59:06.591405Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:59:09.430454Z info ads ADS: new connection for node:3 2026-11-04T01:59:09.430599Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-04T01:59:09.430623Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-04T01:59:13.755409Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:59:58.415367Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T02:00:09.267Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 333 97 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "54c7545a-7637-47c4-a99a-06bd61a5f795" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.178:8000" outbound|54321||e2e-prio-pool-b2b6fa4a-inference-pool-ip-c102c16b.llm.svc.cluster.local 10.133.0.177:52518 10.133.0.177:443 10.132.0.18:54388 - v1-model-routing 2026-11-04T02:00:23.471296Z info ads ADS: "" 3 terminated with stream closed 2026-11-04T02:00:23.779967Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:00:28.928768Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:00:46.247530Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:03:27.803881Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:03:58.228970Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:04:25.607661Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:04:25.607695Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:04:25.607708Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:04:25.607713Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-04T02:04:25.472Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 80 322 135 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "d9027e3b-e83f-466e-a022-e5a751f6a147" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.177:59636 10.133.0.177:443 10.132.0.18:36432 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:04:28.658799Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:04:42.935697Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:04:57.182814Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:04:57.182843Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:04:57.182854Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:04:57.182859Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-04T02:04:57.126Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 56 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "aa1e37c6-a8c9-44f3-bd54-3d3308776bd2" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.177:59636 10.133.0.177:443 10.132.0.18:39054 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:04:58.563615Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:05:11.476196Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:05:16.883036Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T02:05:20.669Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 81 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "c95ccec5-ac6e-4322-b158-60f01029e862" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:43898 10.133.0.177:443 10.132.0.18:37000 - v1-completions-path [2026-11-04T02:05:21.823Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "ae58e788-0618-41b7-b1d6-274073f21917" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:43898 10.133.0.177:443 10.132.0.18:37016 - v1-completions-path 2026-11-04T02:05:36.919246Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:06:25.044372Z info ads ADS: new connection for node:4 2026-11-04T02:06:25.047834Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-04T02:06:25.047868Z info cache read certificate from file resource=file-root:system 2026-11-04T02:06:28.806070Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:06:30.256659Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:06:33.581478Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"152","StreamId":"16316384586683075907"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-04T02:06:33.516Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 65 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "94c12f42-bc76-4182-862f-8f65b3f23d65" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.133.0.177:443 10.132.0.18:40436 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-04T02:06:52.376500Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-04T02:06:52.518444Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:06:52.518481Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:06:52.518492Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:06:52.518497Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T02:06:52.518673Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-04T02:06:52.313Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1527 205 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "8e32eb8e-508f-4708-ad44-3c682a7ad3fe" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.177:34620 10.133.0.177:443 10.132.0.18:49968 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-04T02:07:28.145360Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:07:28.145401Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:07:28.145417Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T02:07:28.145423Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:07:28.145707Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-04T02:07:27.951Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1533 194 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "e5a757f4-43aa-479a-be20-a715842313be" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.177:46734 10.133.0.177:443 10.132.0.18:59956 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-04T02:07:42.349861Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:07:42.349895Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:07:42.349908Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:07:42.349914Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T02:07:42.350105Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-04T02:07:42.229Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1534 120 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "18f112e2-e76b-41a4-ad17-81f5d702750d" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.177:34620 10.133.0.177:443 10.132.0.18:44180 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-04T02:07:47.024330Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:07:47.024361Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:07:47.024371Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:07:47.024375Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T02:07:47.024519Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-04T02:07:46.900Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1532 124 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "79e45304-32a0-4c39-a71a-9ae0c3bf8ef8" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.177:34620 10.133.0.177:443 10.132.0.18:58788 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-04T02:08:27.451144Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:08:30.380588Z info ads ADS: "" 4 terminated 2026-11-04T02:08:30.448947Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:08:31.170109Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:11:50.723495Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-04T02:11:50.723522Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-04T02:11:50.723526Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-04T02:11:50.723529Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-04T02:11:50.670Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 53 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "e7c76e5f-7080-4cb0-b385-fe15499dcf4d" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.177:443 10.132.0.18:55600 - v1-completions-path 2026-11-04T02:12:01.691877Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:12:21.027421Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:12:25.994531Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:12:31.203913Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-04T02:12:31.136Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 68 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "6950c544-d2ef-43a9-a2c6-65ee8ef88a1d" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:50328 10.133.0.177:443 10.132.0.18:50556 - v1-completions-path [2026-11-04T02:12:32.511Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 65 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "7042841e-e862-45b1-82b0-32dfb867692b" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:50328 10.133.0.177:443 10.132.0.18:56560 - v1-completions-path 2026-11-04T02:12:50.727764Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:13:03.703583Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:13:10.135553Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:13:10.135593Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:13:10.135608Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T02:13:10.135613Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T02:13:10.007Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 127 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "b8086e14-4097-41d1-9420-6d937bd23b0c" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.177:34424 10.133.0.177:443 10.132.0.18:53232 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T02:13:12.873Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 57 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "a56c84e3-6981-4f9f-bab8-1e89e15b648c" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:50328 10.133.0.177:443 10.132.0.18:52686 - v1-completions-path 2026-11-04T02:13:35.036680Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:13:46.197614Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:13:51.224519Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T02:14:02.889Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 71 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "36f8e710-9988-4a32-9060-3985fa73760f" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:43898 10.133.0.177:443 10.132.0.18:34516 - v1-completions-path [2026-11-04T02:14:03.159Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 68 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "3ca2f6c9-9f41-4e8e-afc8-f910e3391159" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:43898 10.133.0.177:443 10.132.0.18:34518 - v1-completions-path [2026-11-04T02:14:04.273Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 63 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "d3db1ad0-7174-4e59-be71-d082b8fedce6" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:50328 10.133.0.177:443 10.132.0.18:34536 - v1-completions-path 2026-11-04T02:14:20.916912Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:14:25.863500Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:14:39.967789Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:14:49.650073Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:14:59.510673Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:15:06.302848Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:15:21.704249Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:15:34.387719Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:15:40.886629Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:15:59.495086Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:16:01.459265Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:16:01.459303Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T02:16:01.459317Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T02:16:01.459322Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T02:16:01.345Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 89 331 113 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "b1a94d9d-37d7-4e89-b67c-9c51fbdaff9b" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.25:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.177:34220 10.133.0.177:443 10.132.0.18:37028 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T02:16:05.325153Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:16:17.221396Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:16:43.017870Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:16:53.954362Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:17:12.705931Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:17:20.813736Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:19:50.448647Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:19:56.806382Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:21:24.000160Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:21:41.377589Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:21:45.756683Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:21:58.441666Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:22:15.581591Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:22:22.004288Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:22:41.574671Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:22:52.335214Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:22:58.472251Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:23:25.491432Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:23:29.983084Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:23:35.818529Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T02:23:43.035Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "59ea0eb4-7575-444a-8ad8-e1c962323330" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:50328 10.133.0.177:443 10.132.0.18:60708 - v1-completions-path [2026-11-04T02:23:45.861Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 61 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "d397ad98-4d9c-4ea2-8b14-113925172838" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:43898 10.133.0.177:443 10.132.0.18:60724 - v1-completions-path [2026-11-04T02:23:48.418Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 67 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "9f83ba02-d8e1-401f-a9ec-305d6b50e5f4" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:50328 10.133.0.177:443 10.132.0.18:60740 - v1-completions-path 2026-11-04T02:24:05.099048Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:24:15.978483Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:24:22.450583Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:24:28.294568Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T02:24:33.833Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 62 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "cb751aff-754e-497e-aa80-9377a19c422c" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:50328 10.133.0.177:443 10.132.0.18:39772 - v1-completions-path 2026-11-04T02:24:37.316994Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T02:24:44.655Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 62 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "054644d2-399c-4bd4-9ebf-09b62e2d75a4" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:50328 10.133.0.177:443 10.132.0.18:38180 - v1-completions-path [2026-11-04T02:24:45.022Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 79 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "07d624db-2392-4e2f-8f58-9fc5a2b05bc3" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "10.133.0.33:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.177:50328 10.133.0.177:443 10.132.0.18:38188 - v1-completions-path 2026-11-04T02:25:00.655177Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:25:11.359305Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:25:16.049605Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:25:34.089898Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:25:39.981686Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:25:58.911729Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:26:20.850103Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:26:42.281997Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:27:02.069497Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:27:12.843580Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:27:23.536099Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:27:26.766350Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-04T02:27:34.366853Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:27:45.113970Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T02:27:55.694591Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T02:28:07.856Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 40 - "34.228.250.194,10.132.0.18" "python-requests/2.34.2" "9728901a-ffd3-4971-9f16-6cb9baee6d5c" "maas.apps.aeb4087126d95ff136b4.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.177:443 10.132.0.18:59856 - v1-completions-path