2026-06-02 21:52:38.301 1 storage.initializer INFO [initializer-entrypoint:<module>():16] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')]
2026-06-02 21:52:38.302 1 storage.initializer INFO [kserve_storage.py:download():161] Copying contents of hf://facebook/opt-125m to local
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ac481c8eb05e4d2496fbe076a38a7b4835dd733d.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_f6fe7b5e-814c-4852-9774-9cde0ba8ed7c'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/5HHJ6px3_ZRDOG3OxNZMhuycwOk=.a591333512516f58bf2002045dece909a0ccdb8b.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_e7389d42-6655-4cec-85e8-dfb7d6b110b4'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Xn7B-BWUGOee2Y6hCZtEhtFu4BE=.38c05904caf6e5b9f04ecda5c973d77e6c1da151.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_b982d835-1080-4b81-8853-5d2a4011c108'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_a86a873a-ef8f-4958-9d2d-a311ad5c7c0e'.
Continuing without setting permissions.
{"timestamp":"2026-06-02T21:52:43.864849Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:43.864880Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 2.694877539s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:43.901127Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:43.901142Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 757.132567ms before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:43.962776Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:43.962789Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 38.232108ms before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:44.083203Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:44.083221Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 1.309264859s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:44.134177Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:44.134190Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 611.95641ms before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:49.661241Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:49.661258Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 4.334754363s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:50.396145Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:50.396169Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 4.59788954s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:53.962520Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:53.962538Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 2.347291502s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:54.165710Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:54.165730Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 1.143626082s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:54.503300Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:54.503319Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 482.446878ms before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:54.599827Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:54.599844Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 2.68220955s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:58.364121Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:58.364142Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 2.244145854s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:58.847224Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:58.847246Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 2.419446689s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:58.999925Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:58.999944Z","level":"WARN","fields":{"message":"Retry attempt #2. Sleeping 11.897265327s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:52:59.987849Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:52:59.987870Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 3.166858544s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:53:01.093827Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:53:01.093850Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 1.907725238s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:53:01.312115Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:53:01.312136Z","level":"WARN","fields":{"message":"Retry attempt #2. Sleeping 5.649316729s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:53:05.610273Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:53:05.610296Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 2.672686729s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:53:06.267925Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:53:06.267948Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 1.551339411s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:53:11.962778Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:53:11.962798Z","level":"WARN","fields":{"message":"Retry attempt #3. Sleeping 5.532600992s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:53:12.822126Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:53:12.822150Z","level":"WARN","fields":{"message":"Retry attempt #2. Sleeping 11.154166032s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2026-06-02T21:53:15.900369Z","level":"WARN","fields":{"message":"Status Code: 503. Retrying...","request_id":""},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":194}
{"timestamp":"2026-06-02T21:53:15.900391Z","level":"WARN","fields":{"message":"Retry attempt #3. Sleeping 4.927482207s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/gPcsVCQDYDHk-_n0G9uADl7PXIM=.61c60ec52ed43038fff0fbbd68b080c94b0d94b4c8458dbd65965f9b17631c89.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_e42179c6-71aa-478a-82ab-6569d5e6ed93'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_68b5e9ae-e9f4-4fdc-8b25-431eb898376a'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_2319a7b3-04ce-4c91-90b1-b34cc3b45717'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Q1p2l2BzM1m6P5jKvr8WTq1TUio=.2d74da6615135c58cf3cf9ad4cb11e7c613ff9e55fe658a47ab83b6c8d1174a9.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_ad9e1507-7976-4cb1-a3cc-1c3b1183c3a1'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_ea695c40-3ff7-441f-90ee-1310084c924d'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/a7eHxRFT3OeMBIFg52k2nfj5m7w=.db7090b0c8b34dd957a7e0656c718f978f9203cc874018f37dda44108be5970a.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_09123b89-791f-47a8-b38e-01bc53c704f2'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_b0747fb8-939a-4336-805b-d8f137fbf9e0'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_a8cb8d73-5ede-4650-9707-33a16a251d17'.
Continuing without setting permissions.
2026-06-02 21:53:54.943 1 storage.initializer INFO [kserve_storage.py:download():229] Successfully copied hf://facebook/opt-125m to /mnt/models
2026-06-02 21:53:54.943 1 storage.initializer INFO [kserve_storage.py:download():230] Model downloaded in 76.64062538500002 seconds.
I0602 21:53:56.763846       1 config.go:602] "Configuration:" =<
	{
	  "IP": "10.134.0.51",
	  "PodName": "premium-simulated-simulated-premium-kserve-d8bdbcb5c-vwpvd",
	  "PodNameSpace": "llm",
	  "VllmDevMode": false,
	  "block-size": 16,
	  "data-parallel-rank": -1,
	  "data-parallel-size": 1,
	  "dataset-in-memory": false,
	  "dataset-path": "",
	  "dataset-table-name": "llmd",
	  "dataset-url": "",
	  "default-embedding-dimensions": 384,
	  "ec-transfer-config": "",
	  "enable-kvcache": false,
	  "enable-prefix-caching": false,
	  "enable-request-id-headers": false,
	  "enable-sleep-mode": false,
	  "enforce-eager": false,
	  "event-batch-size": 16,
	  "failure-injection-rate": 0,
	  "failure-types": null,
	  "fake-metrics": null,
	  "fake-metrics-refresh-interval": 100000000,
	  "global-cache-hit-threshold": 0,
	  "hash-seed": "",
	  "inter-token-latency": 0,
	  "inter-token-latency-std-dev": 0,
	  "kv-cache-size": 1024,
	  "kv-cache-transfer-latency": 0,
	  "kv-cache-transfer-latency-std-dev": 0,
	  "kv-cache-transfer-time-per-token": 0,
	  "kv-cache-transfer-time-std-dev": 0,
	  "latency-calculator": "",
	  "lora-modules": null,
	  "max-cpu-loras": 1,
	  "max-loras": 1,
	  "max-model-len": 1024,
	  "max-num-seqs": 5,
	  "max-tool-call-array-param-length": 5,
	  "max-tool-call-integer-param": 100,
	  "max-tool-call-number-param": 100,
	  "max-waiting-queue-length": 1000,
	  "min-tool-call-array-param-length": 1,
	  "min-tool-call-integer-param": 0,
	  "min-tool-call-number-param": 0,
	  "mm-encoder-only": false,
	  "mm-processor-kwargs": "",
	  "mode": "random",
	  "model": "facebook/opt-125m",
	  "object-tool-call-not-required-field-probability": 50,
	  "port": 8000,
	  "prefill-overhead": 0,
	  "prefill-time-per-token": 0,
	  "prefill-time-std-dev": 0,
	  "seed": 1780437236763433000,
	  "self-signed-certs": false,
	  "served-model-name": [
	    "facebook/opt-125m"
	  ],
	  "ssl-certfile": "/var/run/kserve/tls/tls.crt",
	  "ssl-keyfile": "/var/run/kserve/tls/tls.key",
	  "time-factor-under-load": 1,
	  "time-to-first-token": 0,
	  "time-to-first-token-std-dev": 0,
	  "tool-call-not-required-param-probability": 50,
	  "uds-socket-path": "/tmp/tokenizer/tokenizer-uds.socket",
	  "zmq-endpoint": "tcp://127.0.0.1:5557"
	}
 >
I0602 21:53:56.797061       1 tokenizer.go:104] "Model is not a real HF model, using simulated tokenizer" model="facebook/opt-125m"
I0602 21:53:56.801209       1 context.go:138] "No dataset path or URL provided, using random text for responses"
I0602 21:53:56.801282       1 communication.go:49] "Starting communication layer"
I0602 21:53:56.801281       1 simulator.go:188] "Start processing routine"
I0602 21:53:56.801600       1 http_server_tls.go:44] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key"
I0602 21:53:56.801652       1 grpc.go:126] "Server starting" protocol="gRPC" port=8000
I0602 21:53:56.802500       1 http.go:96] "Server starting" protocol="HTTPS" port=8000