Skip to content

[Distributed] Set device_id only for NCCL - #50281

Merged
3outeille merged 2 commits into
huggingface:mainfrom
michaelbenayoun:device_id_only_for_gpus
Nov 3, 2026
Merged

3outeille merged 2 commits into
huggingface:mainfrom
michaelbenayoun:device_id_only_for_gpus

Conversation

@michaelbenayoun

@michaelbenayoun michaelbenayoun commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

🤗 Transformers CI

What does this PR do?

Issue

_ensure_torch_distributed sets device_id for every backend except CPUs. This allows to initialize a process group and tell it right away which device is associated to the current rank. It provides several benefits afterwards.

The issue is that when the DeviceMesh is built, it sees a bound_device_id and builds its subgroups with split_groups, which is not supported by all backends. For example Trainium does not support it:

[rank0]:   File "/home/michaelbenayoun/transformers/.venv/lib/python3.14/site-packages/torch/distributed/c10d_logger.py", line 97, in wrapper
[rank0]:     func_return = func(*args, **kwargs)
[rank0]:   File "/home/michaelbenayoun/transformers/.venv/lib/python3.14/site-packages/torch/distributed/distributed_c10d.py", line 6652, in split_group
[rank0]:     raise RuntimeError(
[rank0]:         "No backend for the parent process group or its backend does not support splitting"
[rank0]:     )
[rank0]: RuntimeError: No backend for the parent process group or its backend does not support splitting
E1102 22:05:05.125000 4719 torch/distributed/elastic/multiprocessing/api.py:1002] failed (exitcode: 1) local_rank: 0 (pid: 4787) of binary: /home/michaelbenayoun/transformers/.venv/bin/py
thon3

Fix

We pass device_id only for the NCCL backend. Other supported backends can be added later on.

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker
ArthurZucker added this pull request to the merge queue Nov 3, 2026
@3outeille
3outeille removed this pull request from the merge queue due to a manual request Nov 3, 2026
@3outeille
3outeille added this pull request to the merge queue Nov 3, 2026
Merged via the queue into huggingface:main with commit 409c349 Nov 3, 2026
124 of 125 checks passed
@michaelbenayoun
michaelbenayoun deleted the device_id_only_for_gpus branch November 3, 2026 23:43
@vasqu vasqu added the for patch Tag issues / labels that should be included in the next patch label Nov 6, 2026
vasqu pushed a commit that referenced this pull request Nov 6, 2026
fix: set device_id only for nccl

Co-authored-by: Ferdinand Mom <47445085+3outeille@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

for patch Tag issues / labels that should be included in the next patch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants