[Kubeflow] Initializing progression tracking
[Kubeflow] Progression tracking enabled
CPU mode forced via GPU_TYPE=cpu
Detected accelerator: CPU (forced)
Using backend: gloo, fp16=False, bf16=False
Distributed: WORLD_SIZE=1, RANK=0, LOCAL_RANK=0
HuggingFace mode: model=distilgpt2, dataset=yahma/alpaca-cleaned
Downloading model from HuggingFace: distilgpt2
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Loading weights:   0%|          | 0/76 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 76/76 [00:00<00:00, 6030.63it/s]
Model downloaded
Downloading dataset from HuggingFace: yahma/alpaca-cleaned

Generating train split:   0%|          | 0/51760 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 51760/51760 [00:00<00:00, 86043.28 examples/s]
Generating train split: 100%|██████████| 51760/51760 [00:00<00:00, 85839.70 examples/s]
Dataset downloaded

Map:   0%|          | 0/80 [00:00<?, ? examples/s]
Map: 100%|██████████| 80/80 [00:00<00:00, 2417.94 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|██████████| 20/20 [00:00<00:00, 1678.70 examples/s]
Starting training on CPU (forced)...

  0%|          | 0/100 [00:00<?, ?it/s][Kubeflow] Metrics server started on port 28080
[transformers] `loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
[rank0]:[W1023 04:12:37.006240488 reducer.cpp:1529] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())

  1%|          | 1/100 [00:03<06:26,  3.90s/it]
  2%|▏         | 2/100 [00:07<06:14,  3.82s/it]
  3%|▎         | 3/100 [00:11<06:07,  3.79s/it]
  4%|▍         | 4/100 [00:15<06:03,  3.79s/it]
  5%|▌         | 5/100 [00:19<06:01,  3.81s/it]
                                               
{'loss': '3.635', 'grad_norm': '7.324', 'learning_rate': '4.8e-05', 'epoch': '0.25'}

  5%|▌         | 5/100 [00:19<06:01,  3.81s/it]
  6%|▌         | 6/100 [00:22<05:57,  3.80s/it]
  7%|▋         | 7/100 [00:26<05:53,  3.81s/it]
  8%|▊         | 8/100 [00:30<05:50,  3.81s/it]
  9%|▉         | 9/100 [00:34<05:46,  3.81s/it]
 10%|█         | 10/100 [00:38<05:43,  3.82s/it]
                                                
{'loss': '2.934', 'grad_norm': '6.635', 'learning_rate': '4.55e-05', 'epoch': '0.5'}

 10%|█         | 10/100 [00:38<05:43,  3.82s/it]
 11%|█         | 11/100 [00:41<05:39,  3.82s/it]
 12%|█▏        | 12/100 [00:45<05:36,  3.83s/it]
 13%|█▎        | 13/100 [00:49<05:32,  3.83s/it]
 14%|█▍        | 14/100 [00:53<05:29,  3.83s/it]
 15%|█▌        | 15/100 [00:57<05:25,  3.83s/it]
                                                
{'loss': '3.058', 'grad_norm': '8.842', 'learning_rate': '4.3e-05', 'epoch': '0.75'}

 15%|█▌        | 15/100 [00:57<05:25,  3.83s/it]
 16%|█▌        | 16/100 [01:01<05:21,  3.82s/it]
 17%|█▋        | 17/100 [01:04<05:17,  3.83s/it]
 18%|█▊        | 18/100 [01:08<05:13,  3.82s/it]
 19%|█▉        | 19/100 [01:12<05:08,  3.81s/it]
 20%|██        | 20/100 [01:16<05:04,  3.80s/it]
                                                
{'loss': '2.909', 'grad_norm': '9', 'learning_rate': '4.05e-05', 'epoch': '1'}

 20%|██        | 20/100 [01:16<05:04,  3.80s/it]/opt/app-root/lib64/python3.12/site-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)


  0%|          | 0/3 [00:00<?, ?it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.11it/s]

100%|██████████| 3/3 [00:02<00:00,  1.12it/s]
                                                
{'eval_loss': '2.689', 'eval_runtime': '4.508', 'eval_samples_per_second': '4.436', 'eval_steps_per_second': '0.665', 'epoch': '1'}

                                             

 20%|██        | 20/100 [01:20<05:04,  3.80s/it]

100%|██████████| 3/3 [00:02<00:00,  1.12it/s]

                                             
 21%|██        | 21/100 [01:24<06:49,  5.18s/it]
 22%|██▏       | 22/100 [01:28<06:12,  4.78s/it]
 23%|██▎       | 23/100 [01:32<05:46,  4.50s/it]
 24%|██▍       | 24/100 [01:36<05:26,  4.30s/it]
 25%|██▌       | 25/100 [01:40<05:11,  4.15s/it]
                                                
{'loss': '2.619', 'grad_norm': '7.332', 'learning_rate': '3.8e-05', 'epoch': '1.25'}

 25%|██▌       | 25/100 [01:40<05:11,  4.15s/it]
 26%|██▌       | 26/100 [01:43<04:59,  4.05s/it]
 27%|██▋       | 27/100 [01:47<04:52,  4.00s/it]
 28%|██▊       | 28/100 [01:51<04:44,  3.96s/it]
 29%|██▉       | 29/100 [01:55<04:38,  3.93s/it]
 30%|███       | 30/100 [01:59<04:33,  3.90s/it]
                                                
{'loss': '2.599', 'grad_norm': '6.068', 'learning_rate': '3.55e-05', 'epoch': '1.5'}

 30%|███       | 30/100 [01:59<04:33,  3.90s/it]
 31%|███       | 31/100 [02:03<04:27,  3.88s/it]
 32%|███▏      | 32/100 [02:06<04:22,  3.87s/it]
 33%|███▎      | 33/100 [02:10<04:18,  3.86s/it]
 34%|███▍      | 34/100 [02:14<04:13,  3.85s/it]
 35%|███▌      | 35/100 [02:18<04:09,  3.84s/it]
                                                
{'loss': '2.585', 'grad_norm': '6.31', 'learning_rate': '3.3e-05', 'epoch': '1.75'}

 35%|███▌      | 35/100 [02:18<04:09,  3.84s/it]
 36%|███▌      | 36/100 [02:22<04:05,  3.83s/it]
 37%|███▋      | 37/100 [02:26<04:01,  3.84s/it]
 38%|███▊      | 38/100 [02:29<03:58,  3.84s/it]
 39%|███▉      | 39/100 [02:33<03:54,  3.84s/it]
 40%|████      | 40/100 [02:37<03:50,  3.84s/it]
                                                
{'loss': '2.692', 'grad_norm': '8.324', 'learning_rate': '3.05e-05', 'epoch': '2'}

 40%|████      | 40/100 [02:37<03:50,  3.84s/it]

  0%|          | 0/3 [00:00<?, ?it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.11it/s]

100%|██████████| 3/3 [00:02<00:00,  1.12it/s]
                                                

                                             
{'eval_loss': '2.569', 'eval_runtime': '4.498', 'eval_samples_per_second': '4.446', 'eval_steps_per_second': '0.667', 'epoch': '2'}

 40%|████      | 40/100 [02:42<03:50,  3.84s/it]

100%|██████████| 3/3 [00:02<00:00,  1.12it/s]

                                             
 41%|████      | 41/100 [02:46<05:07,  5.21s/it]
 42%|████▏     | 42/100 [02:49<04:37,  4.79s/it]
 43%|████▎     | 43/100 [02:53<04:16,  4.50s/it]
 44%|████▍     | 44/100 [02:57<04:01,  4.31s/it]
 45%|████▌     | 45/100 [03:01<03:49,  4.17s/it]
                                                
{'loss': '2.401', 'grad_norm': '8.201', 'learning_rate': '2.8e-05', 'epoch': '2.25'}

 45%|████▌     | 45/100 [03:01<03:49,  4.17s/it]
 46%|████▌     | 46/100 [03:05<03:39,  4.06s/it]
 47%|████▋     | 47/100 [03:09<03:31,  4.00s/it]
 48%|████▊     | 48/100 [03:12<03:25,  3.95s/it]
 49%|████▉     | 49/100 [03:16<03:19,  3.92s/it]
 50%|█████     | 50/100 [03:20<03:14,  3.89s/it]
                                                
{'loss': '2.259', 'grad_norm': '6.23', 'learning_rate': '2.55e-05', 'epoch': '2.5'}

 50%|█████     | 50/100 [03:20<03:14,  3.89s/it]
 51%|█████     | 51/100 [03:24<03:09,  3.88s/it]
 52%|█████▏    | 52/100 [03:28<03:06,  3.88s/it]
 53%|█████▎    | 53/100 [03:32<03:01,  3.86s/it]
 54%|█████▍    | 54/100 [03:35<02:56,  3.85s/it]
 55%|█████▌    | 55/100 [03:39<02:52,  3.84s/it]
                                                
{'loss': '2.476', 'grad_norm': '9.899', 'learning_rate': '2.3e-05', 'epoch': '2.75'}

 55%|█████▌    | 55/100 [03:39<02:52,  3.84s/it]
 56%|█████▌    | 56/100 [03:43<02:49,  3.84s/it]
 57%|█████▋    | 57/100 [03:47<02:45,  3.86s/it]
 58%|█████▊    | 58/100 [03:51<02:41,  3.86s/it]
 59%|█████▉    | 59/100 [03:55<02:37,  3.85s/it]
 60%|██████    | 60/100 [03:59<02:34,  3.86s/it]
                                                
{'loss': '2.334', 'grad_norm': '6.434', 'learning_rate': '2.05e-05', 'epoch': '3'}

 60%|██████    | 60/100 [03:59<02:34,  3.86s/it]

  0%|          | 0/3 [00:00<?, ?it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.11it/s]

100%|██████████| 3/3 [00:02<00:00,  1.12it/s]
                                                

                                             
{'eval_loss': '2.525', 'eval_runtime': '4.509', 'eval_samples_per_second': '4.436', 'eval_steps_per_second': '0.665', 'epoch': '3'}

 60%|██████    | 60/100 [04:03<02:34,  3.86s/it]

100%|██████████| 3/3 [00:02<00:00,  1.12it/s]

                                             
 61%|██████    | 61/100 [04:07<03:23,  5.23s/it]
 62%|██████▏   | 62/100 [04:11<03:03,  4.83s/it]
 63%|██████▎   | 63/100 [04:15<02:47,  4.54s/it]
 64%|██████▍   | 64/100 [04:19<02:35,  4.33s/it]
 65%|██████▌   | 65/100 [04:22<02:26,  4.19s/it]
                                                
{'loss': '2.277', 'grad_norm': '7.247', 'learning_rate': '1.8e-05', 'epoch': '3.25'}

 65%|██████▌   | 65/100 [04:22<02:26,  4.19s/it]
 66%|██████▌   | 66/100 [04:26<02:19,  4.10s/it]
 67%|██████▋   | 67/100 [04:30<02:12,  4.03s/it]
 68%|██████▊   | 68/100 [04:34<02:07,  3.98s/it]
 69%|██████▉   | 69/100 [04:38<02:02,  3.94s/it]
 70%|███████   | 70/100 [04:42<01:57,  3.92s/it]
                                                
{'loss': '2.189', 'grad_norm': '9.545', 'learning_rate': '1.55e-05', 'epoch': '3.5'}

 70%|███████   | 70/100 [04:42<01:57,  3.92s/it]
 71%|███████   | 71/100 [04:46<01:53,  3.91s/it]
 72%|███████▏  | 72/100 [04:50<01:49,  3.90s/it]
 73%|███████▎  | 73/100 [04:53<01:45,  3.90s/it]
 74%|███████▍  | 74/100 [04:57<01:41,  3.89s/it]
 75%|███████▌  | 75/100 [05:01<01:36,  3.88s/it]
                                                
{'loss': '2.168', 'grad_norm': '7.224', 'learning_rate': '1.3e-05', 'epoch': '3.75'}

 75%|███████▌  | 75/100 [05:01<01:36,  3.88s/it]
 76%|███████▌  | 76/100 [05:05<01:32,  3.86s/it]
 77%|███████▋  | 77/100 [05:09<01:29,  3.88s/it]
 78%|███████▊  | 78/100 [05:13<01:25,  3.87s/it]
 79%|███████▉  | 79/100 [05:17<01:21,  3.87s/it]
 80%|████████  | 80/100 [05:20<01:17,  3.87s/it]
                                                
{'loss': '2.23', 'grad_norm': '7.291', 'learning_rate': '1.05e-05', 'epoch': '4'}

 80%|████████  | 80/100 [05:20<01:17,  3.87s/it]

  0%|          | 0/3 [00:00<?, ?it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.08it/s]

100%|██████████| 3/3 [00:02<00:00,  1.10it/s]
                                                

                                             
{'eval_loss': '2.51', 'eval_runtime': '4.554', 'eval_samples_per_second': '4.391', 'eval_steps_per_second': '0.659', 'epoch': '4'}

 80%|████████  | 80/100 [05:25<01:17,  3.87s/it]

100%|██████████| 3/3 [00:02<00:00,  1.10it/s]

                                             
 81%|████████  | 81/100 [05:29<01:39,  5.26s/it]
 82%|████████▏ | 82/100 [05:33<01:27,  4.84s/it]
 83%|████████▎ | 83/100 [05:37<01:17,  4.54s/it]
 84%|████████▍ | 84/100 [05:41<01:09,  4.34s/it]
 85%|████████▌ | 85/100 [05:44<01:02,  4.19s/it]
                                                
{'loss': '2.088', 'grad_norm': '7.95', 'learning_rate': '8e-06', 'epoch': '4.25'}

 85%|████████▌ | 85/100 [05:44<01:02,  4.19s/it]
 86%|████████▌ | 86/100 [05:48<00:57,  4.09s/it]
 87%|████████▋ | 87/100 [05:52<00:52,  4.01s/it]
 88%|████████▊ | 88/100 [05:56<00:47,  3.97s/it]
 89%|████████▉ | 89/100 [06:00<00:43,  3.94s/it]
 90%|█████████ | 90/100 [06:04<00:39,  3.92s/it]
                                                
{'loss': '2.31', 'grad_norm': '7.282', 'learning_rate': '5.5e-06', 'epoch': '4.5'}

 90%|█████████ | 90/100 [06:04<00:39,  3.92s/it]
 91%|█████████ | 91/100 [06:07<00:35,  3.89s/it]
 92%|█████████▏| 92/100 [06:11<00:31,  3.88s/it]
 93%|█████████▎| 93/100 [06:15<00:27,  3.87s/it]
 94%|█████████▍| 94/100 [06:19<00:23,  3.87s/it]
 95%|█████████▌| 95/100 [06:23<00:19,  3.87s/it]
                                                
{'loss': '2.082', 'grad_norm': '6.514', 'learning_rate': '3e-06', 'epoch': '4.75'}

 95%|█████████▌| 95/100 [06:23<00:19,  3.87s/it]
 96%|█████████▌| 96/100 [06:27<00:15,  3.87s/it]
 97%|█████████▋| 97/100 [06:31<00:11,  3.87s/it]
 98%|█████████▊| 98/100 [06:35<00:07,  3.88s/it]
 99%|█████████▉| 99/100 [06:38<00:03,  3.87s/it]
100%|██████████| 100/100 [06:42<00:00,  3.87s/it]
                                                 
{'loss': '2.078', 'grad_norm': '7.42', 'learning_rate': '5e-07', 'epoch': '5'}

100%|██████████| 100/100 [06:42<00:00,  3.87s/it]

  0%|          | 0/3 [00:00<?, ?it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.08it/s]

100%|██████████| 3/3 [00:02<00:00,  1.11it/s]
                                                 

                                             
{'eval_loss': '2.504', 'eval_runtime': '4.531', 'eval_samples_per_second': '4.414', 'eval_steps_per_second': '0.662', 'epoch': '5'}

100%|██████████| 100/100 [06:47<00:00,  3.87s/it]

100%|██████████| 3/3 [00:02<00:00,  1.11it/s]

                                             
                                                 
{'train_runtime': '407.3', 'train_samples_per_second': '0.982', 'train_steps_per_second': '0.246', 'train_loss': '2.496', 'epoch': '5'}

100%|██████████| 100/100 [06:47<00:00,  3.87s/it]
100%|██████████| 100/100 [06:47<00:00,  4.07s/it]
[Kubeflow] Final metrics written to termination message
Training is finished