Summary:
Introduce the torchcomms `_single` collective names on the C++ `c10d::Backend` and migrate the in-tree backends to define them, while keeping the old names fully working for backward compatibility.
`Backend` now declares `all_gather_single`, `all_gather_single_coalesced`, `reduce_scatter_single`, `reduce_scatter_single_coalesced`, and `all_to_all_single` alongside the existing `_allgather_base`, `allgather_into_tensor_coalesced`, `_reduce_scatter_base`, `reduce_scatter_tensor_coalesced`, and `alltoall_base`, which are kept as overridable, forwarding aliases.
Backward compatibility is preserved in both directions: each new method and its old-name alias forward to each other, so a `Backend` subclass may override EITHER name and a caller may invoke EITHER name. The old aliases simply forward to the canonical `_single` method; each canonical method opens with the `C10D_BACKEND_FORWARDING_GUARD()` macro, which installs a function-local thread-local re-entry flag using `__func__` for the message. If a backend overrides neither name the mutual forwarding re-enters the canonical method, the flag trips, and it reports "does not support <method>" instead of recursing forever. As a result, existing callers of the old names (first-party and out-of-tree) and existing out-of-tree backends that override the old names both keep working unchanged. Removing the old overrides outright would have broken out-of-tree backends on the dispatch path, and dropping the old caller-facing names would have broken direct callers; the bidirectional forwarding avoids both.
The old names are intentionally NOT marked `C10_DEPRECATED_MESSAGE` in this change. PyTorch's open-source build compiles the bundled `third_party/torch-xpu-ops` submodule with `-Werror -Wdeprecated`, and its XPU collective op registration (`xccl/Register.cpp`) still calls the old `Backend` names, so a compile-time deprecation turns into a hard build error there. The deprecation will be reintroduced in a follow-up once those callers (and any other out-of-tree backends that are built in-tree) are migrated to the `_single` names and the submodule pin is bumped.
The in-tree backends are migrated to define the new names: `ProcessGroupNCCL`, `ProcessGroupGloo`, `ProcessGroupMPI`, `ProcessGroupUCC`, `ProcessGroupWrapper`, `FakeProcessGroup`, `NCCLXStub`, and the `torch_openreg` (`ProcessGroupOCCL`) test backend. The dispatcher op implementations in `Ops.cpp`, `ProcessGroupWrapper`'s forwarders, and the `Backend` pybind bindings in `init.cpp` use the new names.
The underlying aten / c10d dispatcher op names (`c10d::_allgather_base_`, `c10d::alltoall_base_`, etc.), their schema strings, the `IMPL_*` macro names in `Ops.cpp`, and the `OpType` enum values are intentionally left unchanged for backward compatibility.
Suggested review order: `Backend.hpp` (the new/old method pairs and the bidirectional forwarding + `ForwardingGuard`), then the per-backend subclass renames, then the `Ops.cpp` / `ProcessGroupWrapper` / `init.cpp` call sites.
This diff was authored with the assistance of an AI coding agent (Claude).
Test Plan:
This revision only removes the compile-time deprecation annotations (`C10_DEPRECATED_MESSAGE`), their `-Wdeprecated-declarations` suppressions, and the now-unused `<c10/util/Deprecated.h>` include. The `_single` rename and the bidirectional forwarding/recursion guard are unchanged, so there is no runtime or codegen change -- dropping the annotations only removes deprecation warnings.
Verified by open-source CI on the exported commit: the full libtorch build across platforms, the `linux-noble-xpu-n-py3.10 / build-osdc` job (which compiles the bundled `torch-xpu-ops` `xccl/Register.cpp` with `-Werror -Wdeprecated` and previously failed on `-Werror=deprecated-declarations`), and the `lintrunner-clang` format jobs.
Reviewed By: kapilsh
Differential Revision: D108364288
Pull Request resolved: https://github.com/pytorch/pytorch/pull/187140
Approved by: https://github.com/kapilsh
This PR implements _allgather_base, reduce_scatter, and _reduce_scatter_base in the MPI backend (ProcessGroupMPI), enabling support for Fully Sharded Data Parallel (FSDP) in environments that use MPI for distributed communication.
### Context
As noted in https://github.com/pytorch/pytorch/issues/85628, FSDP currently supports only the NCCL backend. Due to this limitation, FSDP cannot run on legacy HPC environments or clusters that rely on MPI.
By implementing just these three collective operations, we can enable FSDP to work with the MPI backend. These collectives are implemented in a similar manner to existing operations such as allgather.
### Testing
We validated this PR using pytorch/build/bin/ProcessGroupMPITest with OpenMPI, and all tests passed successfully.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/150162
Approved by: https://github.com/H-Huang
Headers under torch/csrc/distributed may be referened with relative path, e.g., "<c10d/...>". However, relative path cannot be gracefully handled by Meta internal build when the NCCL PG is hipified to support AMD/RCCL because the "hipified" header files are generated in other directories. Moreover, using absolute path for header inclusion is the state-of-the-art in most components in Pytorch. Thus, this patch refactors all header paths in torch/csrc/distributed to be absolute.
See D39835774 for more details about Meta internal complication.
**How to test**: commit 9e5d199 removes -I./torch/csrc/distributed in compile options. Thus use it to verify we don't miss any relative path use of torch/csrc/distributed headers.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85780
Approved by: https://github.com/kumpera, https://github.com/huydhn
Headers under torch/csrc/distributed may be referened with relative path, e.g., "<c10d/...>". However, relative path cannot be gracefully handled by Meta internal build when the NCCL PG is hipified to support AMD/RCCL because the "hipified" header files are generated in other directories. Moreover, using absolute path for header inclusion is the state-of-the-art in most components in Pytorch. Thus, this patch refactors all header paths in torch/csrc/distributed to be absolute.
See D39835774 for more details about Meta internal complication.
**How to test**: commit 9e5d199 removes -I./torch/csrc/distributed in compile options. Thus use it to verify we don't miss any relative path use of torch/csrc/distributed headers.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85780
Approved by: https://github.com/kumpera
### Changes
- Move ProcessGroup::Work into its own class and update all the references to it / header includes.
#### Motivation
In the future PRs we will repurpose ProcessGroup to instead contain a list of Backends (ProcessGroupNCCL/Gloo/UCC) and perform dispatching to them based on tensor type. This change is prevent a circular dependency with ProcessGroup depending on Backend and Backend depending on ProcessGroup::Work.
Differential Revision: [D38839212](https://our.internmc.facebook.com/intern/diff/D38839212)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/83680
Approved by: https://github.com/kwen2501
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/60543
Since now c10d is part of libtorch, it would also be nice if the sources lived all in one place.
ghstack-source-id: 132306292
Test Plan: It builds
Reviewed By: cbalioglu
Differential Revision: D29062002
fbshipit-source-id: d9e1301e9d73e1643fa0f0119cd2d618f1ad52e6
Tristan Rice