[Kubeflow] Initializing checkpoint instrumentation
[Kubeflow-node0-rank0] Trainer auto-instrumentation enabled
[Kubeflow] Checkpoint instrumentation enabled
[Kubeflow] Initializing progression tracking
[Kubeflow] Progression tracking enabled
CPU mode forced via GPU_TYPE=cpu
Detected accelerator: CPU (forced)
Using backend: gloo, fp16=False, bf16=False
Distributed: WORLD_SIZE=1, RANK=0, LOCAL_RANK=0
HuggingFace mode: model=distilgpt2, dataset=yahma/alpaca-cleaned
Downloading model from HuggingFace: distilgpt2
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Loading weights:   0%|          | 0/76 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 76/76 [00:00<00:00, 6409.96it/s]
Model downloaded
Downloading dataset from HuggingFace: yahma/alpaca-cleaned

Generating train split:   0%|          | 0/51760 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 51760/51760 [00:00<00:00, 87841.14 examples/s]
Generating train split: 100%|██████████| 51760/51760 [00:00<00:00, 87621.26 examples/s]
Dataset downloaded

Map:   0%|          | 0/80 [00:00<?, ? examples/s]
Map: 100%|██████████| 80/80 [00:00<00:00, 2410.64 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|██████████| 20/20 [00:00<00:00, 1591.71 examples/s]
[Kubeflow-node0-rank0] Applied output_dir: /workspace/checkpoints
[Kubeflow-node0-rank0] Applied save_strategy: epoch
[Kubeflow-node0-rank0] Applied save_total_limit: 3
[Kubeflow-node0-rank0] Auto-injected JIT checkpoint callback
Starting training on CPU (forced)...
[Kubeflow-node0-rank0] Rank 0/1 - Waiting for all ranks before training starts...
[Kubeflow-node0-rank0] Rank 0/1 - All ranks synchronized, proceeding...
[Kubeflow-node0-rank0] JIT checkpoint signal handler registered for SIGTERM
[Kubeflow-node0-rank0] JIT checkpointing enabled

  0%|          | 0/100 [00:00<?, ?it/s][Kubeflow] Metrics server started on port 28080
[transformers] `loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
[rank0]:[W1023 03:39:55.225492719 reducer.cpp:1529] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())

  1%|          | 1/100 [00:03<06:27,  3.92s/it]
  2%|▏         | 2/100 [00:07<06:14,  3.82s/it]
  3%|▎         | 3/100 [00:11<06:07,  3.79s/it]
  4%|▍         | 4/100 [00:15<06:04,  3.80s/it]
  5%|▌         | 5/100 [00:19<06:03,  3.82s/it]
                                               
{'loss': '3.635', 'grad_norm': '7.324', 'learning_rate': '4.8e-05', 'epoch': '0.25'}

  5%|▌         | 5/100 [00:19<06:03,  3.82s/it]
  6%|▌         | 6/100 [00:22<06:00,  3.83s/it]
  7%|▋         | 7/100 [00:26<05:57,  3.84s/it]
  8%|▊         | 8/100 [00:30<05:56,  3.88s/it]
  9%|▉         | 9/100 [00:34<05:52,  3.88s/it]
 10%|█         | 10/100 [00:38<05:49,  3.88s/it]
                                                
{'loss': '2.934', 'grad_norm': '6.635', 'learning_rate': '4.55e-05', 'epoch': '0.5'}

 10%|█         | 10/100 [00:38<05:49,  3.88s/it]
 11%|█         | 11/100 [00:42<05:45,  3.89s/it]
 12%|█▏        | 12/100 [00:46<05:41,  3.88s/it]
 13%|█▎        | 13/100 [00:50<05:38,  3.89s/it]
 14%|█▍        | 14/100 [00:54<05:34,  3.89s/it]
 15%|█▌        | 15/100 [00:58<05:31,  3.90s/it]
                                                
{'loss': '3.058', 'grad_norm': '8.842', 'learning_rate': '4.3e-05', 'epoch': '0.75'}

 15%|█▌        | 15/100 [00:58<05:31,  3.90s/it]
 16%|█▌        | 16/100 [01:01<05:28,  3.91s/it]
 17%|█▋        | 17/100 [01:05<05:25,  3.92s/it]
 18%|█▊        | 18/100 [01:09<05:21,  3.92s/it]
 19%|█▉        | 19/100 [01:13<05:18,  3.93s/it]
 20%|██        | 20/100 [01:17<05:14,  3.93s/it]
                                                
{'loss': '2.909', 'grad_norm': '9', 'learning_rate': '4.05e-05', 'epoch': '1'}

 20%|██        | 20/100 [01:17<05:14,  3.93s/it]/opt/app-root/lib64/python3.12/site-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)


  0%|          | 0/3 [00:00<?, ?it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.11it/s]

100%|██████████| 3/3 [00:02<00:00,  1.11it/s]
                                                

                                             
{'eval_loss': '2.689', 'eval_runtime': '4.53', 'eval_samples_per_second': '4.415', 'eval_steps_per_second': '0.662', 'epoch': '1'}

 20%|██        | 20/100 [01:22<05:14,  3.93s/it]

100%|██████████| 3/3 [00:02<00:00,  1.11it/s]

                                             

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]

Writing model shards: 100%|██████████| 1/1 [00:02<00:00,  2.03s/it]
Writing model shards: 100%|██████████| 1/1 [00:02<00:00,  2.03s/it]
----------------------------------------
Exception occurred during processing of request from ('10.134.0.21', 55632)
Traceback (most recent call last):
  File "/usr/lib64/python3.12/socketserver.py", line 318, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib64/python3.12/socketserver.py", line 349, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib64/python3.12/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib64/python3.12/socketserver.py", line 766, in __init__
    self.handle()
  File "/usr/lib64/python3.12/http/server.py", line 440, in handle
    self.handle_one_request()
  File "/usr/lib64/python3.12/http/server.py", line 428, in handle_one_request
    method()
  File "/opt/app-root/src/4221491235.py", line 1102, in do_GET
    self.wfile.write(payload.encode("utf-8"))
  File "/usr/lib64/python3.12/socketserver.py", line 845, in write
    self._sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe
----------------------------------------

 21%|██        | 21/100 [01:33<09:52,  7.50s/it]
 22%|██▏       | 22/100 [01:37<08:20,  6.41s/it]
 23%|██▎       | 23/100 [01:41<07:14,  5.64s/it]
 24%|██▍       | 24/100 [01:45<06:27,  5.10s/it]
 25%|██▌       | 25/100 [01:48<05:54,  4.73s/it]
                                                
{'loss': '2.619', 'grad_norm': '7.332', 'learning_rate': '3.8e-05', 'epoch': '1.25'}

 25%|██▌       | 25/100 [01:48<05:54,  4.73s/it]
 26%|██▌       | 26/100 [01:52<05:29,  4.46s/it]
 27%|██▋       | 27/100 [01:56<05:12,  4.28s/it]
 28%|██▊       | 28/100 [02:00<04:58,  4.15s/it]
 29%|██▉       | 29/100 [02:04<04:48,  4.06s/it]
 30%|███       | 30/100 [02:08<04:40,  4.01s/it]
                                                
{'loss': '2.599', 'grad_norm': '6.068', 'learning_rate': '3.55e-05', 'epoch': '1.5'}

 30%|███       | 30/100 [02:08<04:40,  4.01s/it]
 31%|███       | 31/100 [02:12<04:33,  3.96s/it]
 32%|███▏      | 32/100 [02:15<04:27,  3.94s/it]
 33%|███▎      | 33/100 [02:19<04:22,  3.92s/it]
 34%|███▍      | 34/100 [02:23<04:17,  3.90s/it]
 35%|███▌      | 35/100 [02:27<04:12,  3.89s/it]
                                                
{'loss': '2.585', 'grad_norm': '6.31', 'learning_rate': '3.3e-05', 'epoch': '1.75'}

 35%|███▌      | 35/100 [02:27<04:12,  3.89s/it]
 36%|███▌      | 36/100 [02:31<04:08,  3.89s/it]
 37%|███▋      | 37/100 [02:35<04:04,  3.88s/it]
 38%|███▊      | 38/100 [02:39<03:59,  3.87s/it]
 39%|███▉      | 39/100 [02:42<03:55,  3.86s/it]
 40%|████      | 40/100 [02:46<03:52,  3.87s/it]
                                                
{'loss': '2.692', 'grad_norm': '8.324', 'learning_rate': '3.05e-05', 'epoch': '2'}

 40%|████      | 40/100 [02:46<03:52,  3.87s/it]/opt/app-root/lib64/python3.12/site-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)


  0%|          | 0/3 [00:00<?, ?it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.11it/s]W1023 03:42:46.354000 1 torch/distributed/elastic/agent/server/api.py:753] Received 15 death signal, shutting down workers
W1023 03:42:46.355000 1 torch/distributed/elastic/multiprocessing/api.py:1028] Sending process 3 closing signal SIGTERM
[Kubeflow-node0-rank0] SIGTERM received, checkpoint will be saved at next safe point


100%|██████████| 3/3 [00:02<00:00,  1.11it/s]
                                                

                                             
{'eval_loss': '2.569', 'eval_runtime': '4.52', 'eval_samples_per_second': '4.425', 'eval_steps_per_second': '0.664', 'epoch': '2'}

 40%|████      | 40/100 [02:51<03:52,  3.87s/it]

100%|██████████| 3/3 [00:02<00:00,  1.11it/s]

                                             

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]

Writing model shards: 100%|██████████| 1/1 [00:02<00:00,  2.02s/it]
Writing model shards: 100%|██████████| 1/1 [00:02<00:00,  2.02s/it]
[Kubeflow-node0-rank0] Starting JIT checkpoint at step 40


Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]

Writing model shards: 100%|██████████| 1/1 [00:02<00:00,  2.09s/it]
Writing model shards: 100%|██████████| 1/1 [00:02<00:00,  2.09s/it]
[Kubeflow-node0-rank0] JIT checkpoint completed at step 40 in node 0 rank 0
/opt/app-root/lib64/python3.12/site-packages/torch/distributed/c10d_logger.py:83: FutureWarning: `torch.distributed.all_gather_into_tensor` is deprecated. Please use `torch.distributed.all_gather_single` instead.
  return func(*args, **kwargs)


  0%|          | 0/3 [00:00<?, ?it/s]

 67%|██████▋   | 2/3 [00:01<00:00,  1.10it/s]

100%|██████████| 3/3 [00:02<00:00,  1.08it/s]
                                                

                                             
{'eval_loss': '2.569', 'eval_runtime': '4.594', 'eval_samples_per_second': '4.354', 'eval_steps_per_second': '0.653', 'epoch': '2'}

 40%|████      | 40/100 [03:12<03:52,  3.87s/it]

100%|██████████| 3/3 [00:02<00:00,  1.08it/s]

                                             
                                                
{'train_runtime': '192.7', 'train_samples_per_second': '2.075', 'train_steps_per_second': '0.519', 'train_loss': '2.909', 'epoch': '2'}

 40%|████      | 40/100 [03:12<03:52,  3.87s/it][Kubeflow-node0-rank0] JIT checkpoint complete. Exiting to avoid redundant operations during graceful shutdown.
Traceback (most recent call last):
  File "/opt/app-root/bin/torchrun", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 367, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/torch/distributed/run.py", line 1028, in main
    run(args)
  File "/opt/app-root/lib64/python3.12/site-packages/torch/distributed/run.py", line 1019, in run
    elastic_launch(
  File "/opt/app-root/lib64/python3.12/site-packages/torch/distributed/launcher/api.py", line 194, in __call__
    return launch_agent(
           ^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/torch/distributed/launcher/api.py", line 374, in launch_agent
    result = agent.run()
             ^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 134, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 745, in run
    result = self._invoke_run(role)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 923, in _invoke_run
    time.sleep(monitor_interval)
  File "/opt/app-root/lib64/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 86, in _terminate_process_handler
    raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 1 got signal: 15