[Kubeflow] Initializing checkpoint instrumentation
[Kubeflow-node0-rank0] WARNING: SSL certificate verification disabled. This should only be used with trusted S3-compatible storage.
[Kubeflow-node0-rank0] Cloud storage configured: s3://training-kubeflow-e2e/checkpoints-TestRhaiS3CheckpointingCPU-1792730236535643868/checkpoints (s3)
[Kubeflow-node0-rank0] Trainer auto-instrumentation enabled
[Kubeflow] Checkpoint instrumentation enabled
CPU mode forced via GPU_TYPE=cpu
Detected accelerator: CPU (forced)
Using backend: gloo, fp16=False, bf16=False
Distributed: WORLD_SIZE=1, RANK=0, LOCAL_RANK=0
HuggingFace mode: model=distilgpt2, dataset=yahma/alpaca-cleaned
Downloading model from HuggingFace: distilgpt2
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Loading weights:   0%|          | 0/76 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 76/76 [00:00<00:00, 5588.39it/s]
Model downloaded
Downloading dataset from HuggingFace: yahma/alpaca-cleaned

Generating train split:   0%|          | 0/51760 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 51760/51760 [00:00<00:00, 86726.75 examples/s]
Generating train split: 100%|██████████| 51760/51760 [00:00<00:00, 86492.73 examples/s]
Dataset downloaded

Map:   0%|          | 0/80 [00:00<?, ? examples/s]
Map: 100%|██████████| 80/80 [00:00<00:00, 2214.33 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|██████████| 20/20 [00:00<00:00, 1664.64 examples/s]
[Kubeflow-node0-rank0] Applied output_dir: /mnt/kubeflow-checkpoints
[Kubeflow-node0-rank0] Applied save_strategy: epoch
[Kubeflow-node0-rank0] Applied save_total_limit: 3
[Kubeflow-node0-rank0] Auto-injected JIT checkpoint callback
[Kubeflow-node0-rank0] No existing or valid checkpoints found in cloud storage. Training will start from scratch.
Starting training on CPU (forced)...
[Kubeflow-node0-rank0] Rank 0/1 - Waiting for all ranks before training starts...
[Kubeflow-node0-rank0] Rank 0/1 - All ranks synchronized, proceeding...
[Kubeflow-node0-rank0] JIT checkpoint signal handler registered for SIGTERM
[Kubeflow-node0-rank0] JIT checkpointing enabled

  0%|          | 0/100 [00:00<?, ?it/s][transformers] `loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
[rank0]:[W1023 04:37:48.319412612 reducer.cpp:1529] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())

  1%|          | 1/100 [00:03<06:34,  3.98s/it]
  2%|▏         | 2/100 [00:07<06:19,  3.87s/it]
  3%|▎         | 3/100 [00:11<06:13,  3.85s/it]
  4%|▍         | 4/100 [00:15<06:08,  3.84s/it]
  5%|▌         | 5/100 [00:19<06:05,  3.85s/it]
                                               
{'loss': '3.635', 'grad_norm': '7.324', 'learning_rate': '4.8e-05', 'epoch': '0.25'}

  5%|▌         | 5/100 [00:19<06:05,  3.85s/it]
  6%|▌         | 6/100 [00:23<06:00,  3.83s/it]
  7%|▋         | 7/100 [00:26<05:55,  3.82s/it]
  8%|▊         | 8/100 [00:30<05:51,  3.82s/it]
  9%|▉         | 9/100 [00:34<05:48,  3.83s/it]
 10%|█         | 10/100 [00:38<05:44,  3.83s/it]
                                                
{'loss': '2.934', 'grad_norm': '6.635', 'learning_rate': '4.55e-05', 'epoch': '0.5'}

 10%|█         | 10/100 [00:38<05:44,  3.83s/it]
 11%|█         | 11/100 [00:42<05:40,  3.83s/it]
 12%|█▏        | 12/100 [00:45<05:35,  3.82s/it]
 13%|█▎        | 13/100 [00:49<05:31,  3.81s/it]
 14%|█▍        | 14/100 [00:53<05:27,  3.81s/it]
 15%|█▌        | 15/100 [00:57<05:24,  3.81s/it]
                                                
{'loss': '3.058', 'grad_norm': '8.842', 'learning_rate': '4.3e-05', 'epoch': '0.75'}

 15%|█▌        | 15/100 [00:57<05:24,  3.81s/it]
 16%|█▌        | 16/100 [01:01<05:20,  3.81s/it]
 17%|█▋        | 17/100 [01:05<05:16,  3.81s/it]
 18%|█▊        | 18/100 [01:08<05:11,  3.80s/it]
 19%|█▉        | 19/100 [01:12<05:08,  3.81s/it]
 20%|██        | 20/100 [01:16<05:05,  3.82s/it]
                                                
{'loss': '2.909', 'grad_norm': '9', 'learning_rate': '4.05e-05', 'epoch': '1'}

 20%|██        | 20/100 [01:16<05:05,  3.82s/it]/opt/app-root/lib64/python3.12/site-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)


  0%|          | 0/3 [00:00<?, ?it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.11it/s]

100%|██████████| 3/3 [00:02<00:00,  1.12it/s]
                                                

                                             
{'eval_loss': '2.689', 'eval_runtime': '4.465', 'eval_samples_per_second': '4.479', 'eval_steps_per_second': '0.672', 'epoch': '1'}

 20%|██        | 20/100 [01:20<05:05,  3.82s/it]

100%|██████████| 3/3 [00:02<00:00,  1.12it/s]

                                             

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]

Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  7.34it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  7.33it/s]
[Kubeflow-node0-rank0] Moving checkpoint to staging: checkpoint-20
[Kubeflow-node0-rank0] Background upload worker started
[Kubeflow-node0-rank0] Queued checkpoint for async upload: checkpoint-20
[Kubeflow-node0-rank0] Starting parallel upload to S3: checkpoint-20
[Kubeflow-node0-rank0] Progress: 0.0/940.9 MB (0%)
[Kubeflow-node0-rank0] Progress: 0.0/940.9 MB (0%)

 21%|██        | 21/100 [01:29<08:35,  6.53s/it]
 22%|██▏       | 22/100 [01:33<07:42,  5.94s/it][Kubeflow-node0-rank0] Progress: 3.4/940.9 MB (0%)
[Kubeflow-node0-rank0] Progress: 315.9/940.9 MB (33%)
[Kubeflow-node0-rank0] Progress: 940.9/940.9 MB (100%)
[Kubeflow-node0-rank0] Upload complete: checkpoint-20
[Kubeflow-node0-rank0] Deleted local staging checkpoint: checkpoint-20

 23%|██▎       | 23/100 [01:38<07:08,  5.56s/it]
 24%|██▍       | 24/100 [01:42<06:24,  5.06s/it]
 25%|██▌       | 25/100 [01:46<05:53,  4.71s/it]
                                                
{'loss': '2.619', 'grad_norm': '7.332', 'learning_rate': '3.8e-05', 'epoch': '1.25'}

 25%|██▌       | 25/100 [01:46<05:53,  4.71s/it]
 26%|██▌       | 26/100 [01:50<05:29,  4.46s/it]
 27%|██▋       | 27/100 [01:54<05:12,  4.28s/it]
 28%|██▊       | 28/100 [01:57<04:59,  4.16s/it]
 29%|██▉       | 29/100 [02:01<04:48,  4.07s/it]
 30%|███       | 30/100 [02:05<04:40,  4.01s/it]
                                                
{'loss': '2.599', 'grad_norm': '6.068', 'learning_rate': '3.55e-05', 'epoch': '1.5'}

 30%|███       | 30/100 [02:05<04:40,  4.01s/it]
 31%|███       | 31/100 [02:09<04:33,  3.97s/it]
 32%|███▏      | 32/100 [02:13<04:28,  3.95s/it]
 33%|███▎      | 33/100 [02:17<04:23,  3.93s/it]
 34%|███▍      | 34/100 [02:21<04:17,  3.91s/it]
 35%|███▌      | 35/100 [02:25<04:13,  3.90s/it]
                                                
{'loss': '2.585', 'grad_norm': '6.31', 'learning_rate': '3.3e-05', 'epoch': '1.75'}

 35%|███▌      | 35/100 [02:25<04:13,  3.90s/it]
 36%|███▌      | 36/100 [02:28<04:09,  3.89s/it]
 37%|███▋      | 37/100 [02:32<04:04,  3.87s/it]
 38%|███▊      | 38/100 [02:36<04:00,  3.87s/it]
 39%|███▉      | 39/100 [02:40<03:55,  3.86s/it]
 40%|████      | 40/100 [02:44<03:51,  3.86s/it]
                                                
{'loss': '2.692', 'grad_norm': '8.324', 'learning_rate': '3.05e-05', 'epoch': '2'}

 40%|████      | 40/100 [02:44<03:51,  3.86s/it]/opt/app-root/lib64/python3.12/site-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)


  0%|          | 0/3 [00:00<?, ?it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.10it/s]

100%|██████████| 3/3 [00:02<00:00,  1.11it/s]
                                                

                                             
{'eval_loss': '2.569', 'eval_runtime': '4.494', 'eval_samples_per_second': '4.451', 'eval_steps_per_second': '0.668', 'epoch': '2'}

 40%|████      | 40/100 [02:48<03:51,  3.86s/it]

100%|██████████| 3/3 [00:02<00:00,  1.11it/s]

                                             

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]

Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  7.42it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  7.41it/s]
W1023 04:40:37.671000 1 torch/distributed/elastic/agent/server/api.py:753] Received 15 death signal, shutting down workers
W1023 04:40:37.672000 1 torch/distributed/elastic/multiprocessing/api.py:1028] Sending process 3 closing signal SIGTERM
[Kubeflow-node0-rank0] SIGTERM received, checkpoint will be saved at next safe point
[Kubeflow-node0-rank0] Moving checkpoint to staging: checkpoint-40
[Kubeflow-node0-rank0] Queued checkpoint for async upload: checkpoint-40[Kubeflow-node0-rank0] Starting parallel upload to S3: checkpoint-40

[Kubeflow-node0-rank0] Starting JIT checkpoint at step 40


Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]

Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  3.35it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  3.35it/s]
[Kubeflow-node0-rank0] Progress: 0.0/940.9 MB (0%)
[Kubeflow-node0-rank0] Moving checkpoint to staging: checkpoint-40
[Kubeflow-node0-rank0] Progress: 3.4/940.9 MB (0%)
[Kubeflow-node0-rank0] Queued checkpoint for async upload: checkpoint-40
[Kubeflow-node0-rank0] JIT checkpoint completed at step 40 in node 0 rank 0
[Kubeflow-node0-rank0] Progress: 315.9/940.9 MB (33%)
/opt/app-root/lib64/python3.12/site-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)


  0%|          | 0/3 [00:00<?, ?it/s][Kubeflow-node0-rank0] Upload complete: checkpoint-40
[Kubeflow-node0-rank0] Deleted local staging checkpoint: checkpoint-40
[Kubeflow-node0-rank0] Starting parallel upload to S3: checkpoint-40
[Kubeflow-node0-rank0] Upload complete: checkpoint-40
[Kubeflow-node0-rank0] Warning: Failed to delete local staging checkpoint checkpoint-40: [Errno 2] No such file or directory: '/mnt/kubeflow-checkpoints/cloud-upload-staging/checkpoint-40'. Upload succeeded, but local cleanup failed. Staging data will be cleaned up when the pod terminates.


 67%|██████▋   | 2/3 [00:02<00:01,  1.08s/it]

100%|██████████| 3/3 [00:03<00:00,  1.02s/it]
                                                

                                             
{'eval_loss': '2.569', 'eval_runtime': '5.565', 'eval_samples_per_second': '3.594', 'eval_steps_per_second': '0.539', 'epoch': '2'}

 40%|████      | 40/100 [03:07<03:51,  3.86s/it]

100%|██████████| 3/3 [00:03<00:00,  1.02s/it]

                                             
                                                
{'train_runtime': '187.2', 'train_samples_per_second': '2.137', 'train_steps_per_second': '0.534', 'train_loss': '2.909', 'epoch': '2'}

 40%|████      | 40/100 [03:07<03:51,  3.86s/it][Kubeflow-node0-rank0] Waiting for background uploads to complete...