2026-06-10 11:50:43.703 1 storage.initializer INFO [initializer-entrypoint:<module>():16] Initializing, args: (src_uri, dest_path): [('hf://sshleifer/tiny-gpt2', '/mnt/models')]
2026-06-10 11:50:43.703 1 storage.initializer INFO [kserve_storage.py:download():161] Copying contents of hf://sshleifer/tiny-gpt2 to local
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ae8c63daedbd4206d7d40126955d4e6ab1c80f8f.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_a8d08e3a-c826-4f3f-8060-04b1654d875e'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.2c81a6c4c984e95a45338c64a7445c1f0f88077f.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_5fbc96a7-6a86-4c31-b08b-0d1ab378c8e8'.
Continuing without setting permissions.
{"timestamp":"2026-06-10T11:51:34.167402Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://us.aws.cdn.hf.co/xorbs/default/25327c62b388c329c0a8956b9eaaf146ba4141c4b3a02f20634e52aa0be9a7ba?repo_id=621ffdc136468d709f18090c&X-Xet-Session-Id=01KTRNXPHPSHJ4JVSH2CQ2SYJR&user_id=public&Expires=1781095844&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hvcmJzL2RlZmF1bHQvMjUzMjdjNjJiMzg4YzMyOWMwYTg5NTZiOWVhYWYxNDZiYTQxNDFjNGIzYTAyZjIwNjM0ZTUyYWEwYmU5YTdiYVxcP3JlcG9faWQ9NjIxZmZkYzEzNjQ2OGQ3MDlmMTgwOTBjJlgtWGV0LVNlc3Npb24tSWQ9MDFLVFJOWFBIUFNISjRKVlNIMkNRMlNZSlImdXNlcl9pZD1wdWJsaWMiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkVwb2NoVGltZSI6MTc4MTA5NTg0NH0sIkJ5dGVSYW5nZSI6eyJFeHBlY3RlZEhlYWRlciI6ImJ5dGVzPTAtMzkxMDU0In19fV19&Signature=MEUCIHcaNPdHoszbeeNz%7EFSUYUZKUqsH6Be5Xz%7EGbkn6p%7EDUAiEA0gztfpX4WjYAhM4gVwDFxaYaJ5E%7E-VAWFqupXmbQhOw_&Key-Pair-Id=01KAYHXK2CBJSW0YZTMNXK9W1M\", source: hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(5), cause: None }, X509VerifyResult { code: 0, error: \"ok\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2026-06-10T11:51:34.167453Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 815.290644ms before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/gPcsVCQDYDHk-_n0G9uADl7PXIM=.3cd4987249615ca870f2c18e657a62f0962aecad0eac790dd89362227310fd30.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_5894e954-6368-4e4d-b841-34c79a46abc4'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_9ccaec71-f77b-45f5-9ba0-c4d2ca07cb0b'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Q1p2l2BzM1m6P5jKvr8WTq1TUio=.b706b24034032bdfe765ded5ab6403d201d295a995b790cb24c74becca5c04e6.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_09ec88be-19fc-4fd7-b163-23130cf96b73'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.817762d631ad6f9c799f6b9dc713c46420e65546.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_480e3ced-5c4c-4ba2-bb63-b33fac618cb5'.
Continuing without setting permissions.
{"timestamp":"2026-06-10T11:52:25.390360Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://us.aws.cdn.hf.co/xorbs/default/25327c62b388c329c0a8956b9eaaf146ba4141c4b3a02f20634e52aa0be9a7ba?repo_id=621ffdc136468d709f18090c&user_id=public&X-Xet-Session-Id=01KTRNZ8MJYQJP8N18GE5CB6NM&Expires=1781095895&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly91cy5hd3MuY2RuLmhmLmNvL3hvcmJzL2RlZmF1bHQvMjUzMjdjNjJiMzg4YzMyOWMwYTg5NTZiOWVhYWYxNDZiYTQxNDFjNGIzYTAyZjIwNjM0ZTUyYWEwYmU5YTdiYVxcP3JlcG9faWQ9NjIxZmZkYzEzNjQ2OGQ3MDlmMTgwOTBjJnVzZXJfaWQ9cHVibGljJlgtWGV0LVNlc3Npb24tSWQ9MDFLVFJOWjhNSllRSlA4TjE4R0U1Q0I2Tk0iLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkVwb2NoVGltZSI6MTc4MTA5NTg5NX0sIkJ5dGVSYW5nZSI6eyJFeHBlY3RlZEhlYWRlciI6ImJ5dGVzPTgzMDgxMy0xMjQ1NzY1In19fV19&Signature=MEUCIDg8G6q28zZeCocTVAq8YCe%7Ey7VLofm0AsIVINxG1yEMAiEAuEAVa7rwQ01AuAdL6S4qBKMvozPbTNTljMChvMauspY_&Key-Pair-Id=01KAYHXK2CBJSW0YZTMNXK9W1M\", source: hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(5), cause: None }, X509VerifyResult { code: 0, error: \"ok\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2026-06-10T11:52:25.390403Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 1.619853851s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/a7eHxRFT3OeMBIFg52k2nfj5m7w=.09166c56d46ec8eca5f1e46e4e4a62265fca5975a70a2b263c1795291a685cf7.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_3e78a562-99c3-4f27-b937-b9ccee62e742'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.be4d21d94f3b4687e5a54d84bf6ab46ed0f8defd.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_99ce1589-b44a-471e-b478-f36e9aa18cb9'.
Continuing without setting permissions.
Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.b00361fece0387ca34b4b8b8539ed830d644dbeb.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_0344dfed-f777-4884-ad22-d87d64a139bd'.
Continuing without setting permissions.
2026-06-10 11:52:27.104 1 storage.initializer INFO [kserve_storage.py:download():229] Successfully copied hf://sshleifer/tiny-gpt2 to /mnt/models
2026-06-10 11:52:27.104 1 storage.initializer INFO [kserve_storage.py:download():230] Model downloaded in 103.4003687180001 seconds.
I0610 11:52:27.928132       1 config.go:602] "Configuration:" =<
	{
	  "IP": "10.134.0.29",
	  "PodName": "e2e-trlp-test-simulated-kserve-68bb6c58d9-rn29w",
	  "PodNameSpace": "llm",
	  "VllmDevMode": false,
	  "block-size": 16,
	  "data-parallel-rank": -1,
	  "data-parallel-size": 1,
	  "dataset-in-memory": false,
	  "dataset-path": "",
	  "dataset-table-name": "llmd",
	  "dataset-url": "",
	  "default-embedding-dimensions": 384,
	  "ec-transfer-config": "",
	  "enable-kvcache": false,
	  "enable-prefix-caching": false,
	  "enable-request-id-headers": false,
	  "enable-sleep-mode": false,
	  "enforce-eager": false,
	  "event-batch-size": 16,
	  "failure-injection-rate": 0,
	  "failure-types": null,
	  "fake-metrics": null,
	  "fake-metrics-refresh-interval": 100000000,
	  "global-cache-hit-threshold": 0,
	  "hash-seed": "",
	  "inter-token-latency": 0,
	  "inter-token-latency-std-dev": 0,
	  "kv-cache-size": 1024,
	  "kv-cache-transfer-latency": 0,
	  "kv-cache-transfer-latency-std-dev": 0,
	  "kv-cache-transfer-time-per-token": 0,
	  "kv-cache-transfer-time-std-dev": 0,
	  "latency-calculator": "",
	  "lora-modules": null,
	  "max-cpu-loras": 1,
	  "max-loras": 1,
	  "max-model-len": 1024,
	  "max-num-seqs": 5,
	  "max-tool-call-array-param-length": 5,
	  "max-tool-call-integer-param": 100,
	  "max-tool-call-number-param": 100,
	  "max-waiting-queue-length": 1000,
	  "min-tool-call-array-param-length": 1,
	  "min-tool-call-integer-param": 0,
	  "min-tool-call-number-param": 0,
	  "mm-encoder-only": false,
	  "mm-processor-kwargs": "",
	  "mode": "random",
	  "model": "test/e2e-trlp-test-model",
	  "object-tool-call-not-required-field-probability": 50,
	  "port": 8000,
	  "prefill-overhead": 0,
	  "prefill-time-per-token": 0,
	  "prefill-time-std-dev": 0,
	  "seed": 1781092347927696600,
	  "self-signed-certs": false,
	  "served-model-name": [
	    "test/e2e-trlp-test-model"
	  ],
	  "ssl-certfile": "/var/run/kserve/tls/tls.crt",
	  "ssl-keyfile": "/var/run/kserve/tls/tls.key",
	  "time-factor-under-load": 1,
	  "time-to-first-token": 0,
	  "time-to-first-token-std-dev": 0,
	  "tool-call-not-required-param-probability": 50,
	  "uds-socket-path": "/tmp/tokenizer/tokenizer-uds.socket",
	  "zmq-endpoint": "tcp://127.0.0.1:5557"
	}
 >
I0610 11:52:27.956973       1 tokenizer.go:104] "Model is not a real HF model, using simulated tokenizer" model="test/e2e-trlp-test-model"
I0610 11:52:27.963638       1 context.go:138] "No dataset path or URL provided, using random text for responses"
I0610 11:52:27.963721       1 communication.go:49] "Starting communication layer"
I0610 11:52:27.963823       1 simulator.go:188] "Start processing routine"
I0610 11:52:27.964040       1 http_server_tls.go:44] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key"
I0610 11:52:27.964071       1 grpc.go:126] "Server starting" protocol="gRPC" port=8000
I0610 11:52:27.964653       1 http.go:96] "Server starting" protocol="HTTPS" port=8000