# Bug on multi-gpu trainer with accelerate

**URL:** <https://discuss.huggingface.co/t/bug-on-multi-gpu-trainer-with-accelerate/141592>\
**Category:** 🤗Accelerate\
**Created:** [February 18, 2025, 1:21pm UTC](https://discuss.huggingface.co/t/bug-on-multi-gpu-trainer-with-accelerate/141592 "2025-02-18T13:21:04Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![zzzz567](https://avatars.discourse-cdn.com/v4/letter/z/bbce88/32.png) [@zzzz567](https://discuss.huggingface.co/u/zzzz567)\
**Post date:** [February 18, 2025, 1:21pm UTC](https://discuss.huggingface.co/t/bug-on-multi-gpu-trainer-with-accelerate/141592/1 "2025-02-18T13:21:04Z")

</div>

I am running the sample object\_detection [script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/object-detection/run_object_detection_no_trainer.py#L177) from hugging face on 4 gpus using `accelerate launch`.

As soon as the `evaluation_loop` is ran after the end of epoch 1. It fails at the `metrics = metric.compute()` step with the following error:

```auto
[rank3]: work = group.allgather([tensor_list], [tensor])
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: RuntimeError: No backend type associated with device type cpu

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 18, 2025, 2:46pm UTC](https://discuss.huggingface.co/t/bug-on-multi-gpu-trainer-with-accelerate/141592/2 "2025-02-18T14:46:19Z")

</div>

Possibly unresolved issue…

> <https://github.com/Lightning-AI/torchmetrics/issues/2477>
>
> \## 🐛 Bug
> Metrics (predefined in library and custom implementations) using conca…tenation \`dist\_reduce\_fx="cat"\` and CPU computation \`compute\_on\_cpu=True\` raise an error when training in multiple GPUs (\`ddp\`). The concrete error is \`RuntimeError: No backend type associated with device type cpu\`. 
> 
> \### To Reproduce
> 
> 
> 
> \*\*Code sample:\*\*
> 
> \`\`\`py
> import torch
> from lightning import Trainer, LightningModule
> from torch.utils.data import DataLoader
> from torchmetrics import AUROC
> 
> 
> class LitModel(LightningModule):
> def \_\_init\_\_(self) -\> None:
> super().\_\_init\_\_()
> self.layer = torch.nn.Linear(1, 1)
> self.auroc = AUROC(task="binary", compute\_on\_cpu=True)
> 
> def training\_step(self, x):
> preds = torch.tensor(\[0.13, 0.26, 0.08, 0.19, 0.34\]).cuda()
> target = torch.tensor(\[0, 0, 1, 1, 1\]).cuda()
> self.auroc(preds, target)
> self.log("train\_auroc", self.auroc, on\_step=True, on\_epoch=True)
> loss = self.layer(x).mean()
> return loss
> 
> def configure\_optimizers(self):
> return torch.optim.SGD(self.parameters(), lr=0.1)
> 
> def train\_dataloader(self):
> return DataLoader(torch.randn(32, 1), batch\_size=1)
> \`\`\`
> 
> \*\*Stacktrace\*\*
> 
> \`\`\`
> Traceback (most recent call last):
> File "/home/ruben/Documents/PhD/Research/Topological Deep Learning/lightning/pythonProject/main.py", line 35, in \<module\>
> main()
> File "/home/ruben/Documents/PhD/Research/Topological Deep Learning/lightning/pythonProject/main.py", line 31, in main
> trainer.fit(model)
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 544, in fit
> call.\_call\_and\_handle\_interrupt(
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/call.py", line 43, in \_call\_and\_handle\_interrupt
> return trainer.strategy.launcher.launch(trainer\_fn, \*args, trainer=trainer, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/strategies/launchers/subprocess\_script.py", line 105, in launch
> return function(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 580, in \_fit\_impl
> self.\_run(model, ckpt\_path=ckpt\_path)
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 987, in \_run
> results = self.\_run\_stage()
> ^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 1033, in \_run\_stage
> self.fit\_loop.run()
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/loops/fit\_loop.py", line 206, in run
> self.on\_advance\_end()
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/loops/fit\_loop.py", line 376, in on\_advance\_end
> call.\_call\_callback\_hooks(trainer, "on\_train\_epoch\_end", monitoring\_callbacks=False)
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/call.py", line 208, in \_call\_callback\_hooks
> fn(trainer, trainer.lightning\_module, \*args, \*\*kwargs)
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/callbacks/progress/tqdm\_progress.py", line 281, in on\_train\_epoch\_end
> self.train\_progress\_bar.set\_postfix(self.get\_metrics(trainer, pl\_module))
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/callbacks/progress/progress\_bar.py", line 198, in get\_metrics
> pbar\_metrics = trainer.progress\_bar\_metrics
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 1651, in progress\_bar\_metrics
> return self.\_logger\_connector.progress\_bar\_metrics
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/connectors/logger\_connector/logger\_connector.py", line 253, in progress\_bar\_metrics
> metrics = self.metrics\["pbar"\]
> ^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/connectors/logger\_connector/logger\_connector.py", line 234, in metrics
> return self.trainer.\_results.metrics(on\_step)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/connectors/logger\_connector/result.py", line 483, in metrics
> value = self.\_get\_cache(result\_metric, on\_step)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/connectors/logger\_connector/result.py", line 447, in \_get\_cache
> result\_metric.compute()
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/connectors/logger\_connector/result.py", line 289, in wrapped\_func
> self.\_computed = compute(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning/pytorch/trainer/connectors/logger\_connector/result.py", line 254, in compute
> return self.value.compute()
> ^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/torchmetrics/metric.py", line 611, in wrapped\_func
> with self.sync\_context(
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/contextlib.py", line 137, in \_\_enter\_\_
> return next(self.gen)
> ^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/torchmetrics/metric.py", line 582, in sync\_context
> self.sync(
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/torchmetrics/metric.py", line 531, in sync
> self.\_sync\_dist(dist\_sync\_fn, process\_group=process\_group)
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/torchmetrics/metric.py", line 435, in \_sync\_dist
> output\_dict = apply\_to\_collection(
> ^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning\_utilities/core/apply\_func.py", line 72, in apply\_to\_collection
> return \_apply\_to\_collection\_slow(
> ^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning\_utilities/core/apply\_func.py", line 104, in \_apply\_to\_collection\_slow
> v = \_apply\_to\_collection\_slow(
> ^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning\_utilities/core/apply\_func.py", line 125, in \_apply\_to\_collection\_slow
> v = \_apply\_to\_collection\_slow(
> ^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/lightning\_utilities/core/apply\_func.py", line 96, in \_apply\_to\_collection\_slow
> return function(data, \*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/torchmetrics/utilities/distributed.py", line 127, in gather\_all\_tensors
> torch.distributed.all\_gather(local\_sizes, local\_size, group=group)
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/torch/distributed/c10d\_logger.py", line 47, in wrapper
> return func(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^
> File "/home/ruben/miniconda3/envs/sct/lib/python3.11/site-packages/torch/distributed/distributed\_c10d.py", line 2808, in all\_gather
> work = group.allgather(\[tensor\_list\], \[tensor\])
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> RuntimeError: No backend type associated with device type cpu
> \`\`\`
> 
> \### Expected behavior
> 
> Metric is computed properly merging the different lists in the differents processes in multi GPU training scenarios.
> 
> \### Environment
> 
> \- TorchMetrics version 1.3.2. Installed using \`pip\`
> \- Python & PyTorch Version: 3.11 and 2.1.2, respectively.
> \- Any other relevant information such as OS (e.g., Linux): Ubuntu 23.10
> 
> \### Additional context
> 
> Related bug in PyTorch Lightning
> 
> https://github.com/Lightning-AI/pytorch-lightning/issues/18803

---

<div class="post-metadata">

**Author:** ![zzzz567](https://avatars.discourse-cdn.com/v4/letter/z/bbce88/32.png) [@zzzz567](https://discuss.huggingface.co/u/zzzz567)\
**Post date:** [February 18, 2025, 3:18pm UTC](https://discuss.huggingface.co/t/bug-on-multi-gpu-trainer-with-accelerate/141592/3 "2025-02-18T15:18:30Z")

</div>

Thank you @John6666 ! Yes I saw that issue. It seems to be a common problem. The evaluation phase within each epoch is quite fiddly and even with using `gather` I am not seeing the correct number of records per batches

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 18, 2025, 3:28pm UTC](https://discuss.huggingface.co/t/bug-on-multi-gpu-trainer-with-accelerate/141592/4 "2025-02-18T15:28:48Z")

</div>

I thought there might be an issue with gather or allgather, but it doesn’t seem to be the case for this problem.

Since it’s limited to multi-GPU, the number of users is limited, so there may not be many reports, and there may be many potential unresolved bugs.  
I can’t find anything like that on the github for the accelerate library either…

> **[huggingface/transformers](https://github.com/huggingface/transformers/issues)**
>
> 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - huggingface/transformers

---

<div class="post-metadata">

**Author:** ![zzzz567](https://avatars.discourse-cdn.com/v4/letter/z/bbce88/32.png) [@zzzz567](https://discuss.huggingface.co/u/zzzz567)\
**Post date:** [February 18, 2025, 3:43pm UTC](https://discuss.huggingface.co/t/bug-on-multi-gpu-trainer-with-accelerate/141592/5 "2025-02-18T15:43:09Z")

</div>

I am now using this [fix](https://github.com/huggingface/transformers/issues/32525#issuecomment-2283092334). The script is now running fully however I have no way of checking if ` torchmetrics.Metric` did use all the records in my validation set. Any ideas @John6666?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 18, 2025, 3:48pm UTC](https://discuss.huggingface.co/t/bug-on-multi-gpu-trainer-with-accelerate/141592/6 "2025-02-18T15:48:26Z")

</div>

It’s a primitive and dirty method, but could you override some function and insert a print statement or logger?  
Since it’s Python, you could also modify the code of the library itself…

---

<div class="post-metadata">

**Author:** ![zzzz567](https://avatars.discourse-cdn.com/v4/letter/z/bbce88/32.png) [@zzzz567](https://discuss.huggingface.co/u/zzzz567)\
**Post date:** [February 18, 2025, 4:01pm UTC](https://discuss.huggingface.co/t/bug-on-multi-gpu-trainer-with-accelerate/141592/7 "2025-02-18T16:01:21Z")

</div>

Found a solution by creating another torchmetrics using SumMetric()