we want to make it default to replace nccl.
> **AI-generated description:**
>
> nccl2 always distributed one rank-zero unique ID and called ncclCommInitRankConfig, making communicator bootstrap a single-root bottleneck after legacy ProcessGroupNCCL adopted scalable initialization.
>
> Honor TORCH_NCCL_RANKS_PER_ROOT with the same 128-rank default and balanced-root assignment as legacy NCCL. Each root publishes one ID through the prefixed store, every rank fetches the complete ID set, and multi-root groups initialize through ncclCommInitRankScalable while single-root groups retain the existing path. Cover both an even four-rank layout and an uneven three-rank layout with root and non-root participants.
>
> Test Plan:
>
> ```
> .venv/bin/spin quickfix -- --skip CLANGTIDY
> .venv/bin/spin quicklint -- --skip CLANGTIDY
> ```
>
> ```
> BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 USE_FLASH_ATTENTION=0 USE_MEM_EFF_ATTENTION=0 USE_DISTRIBUTED=1 USE_CUDA=1 USE_NCCL_EP=1 USE_SYSTEM_NCCL=1 .venv/bin/pip install -e . -v --no-build-isolation
> ```
>
> ```
> uv run --no-sync python test/distributed/test_c10d_nccl2.py ProcessGroupNCCL2ScalableInitTest.test_collective_with_scalable_init ProcessGroupNCCL2UnevenScalableInitTest.test_collective_with_scalable_init
> ```
>
> Authored with the assistance of an AI coding agent.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/191983
Approved by: https://github.com/kapilsh
ghstack dependencies: #191978, #191979
we want to make it default to replace nccl.
> **AI-generated description:**
>
> nccl2 always distributed one rank-zero unique ID and called ncclCommInitRankConfig, making communicator bootstrap a single-root bottleneck after legacy ProcessGroupNCCL adopted scalable initialization.
>
> Honor TORCH_NCCL_RANKS_PER_ROOT with the same 128-rank default and balanced-root assignment as legacy NCCL. Each root publishes one ID through the prefixed store, every rank fetches the complete ID set, and multi-root groups initialize through ncclCommInitRankScalable while single-root groups retain the existing path. Cover both an even four-rank layout and an uneven three-rank layout with root and non-root participants.
>
> Test Plan:
>
> ```
> .venv/bin/spin quickfix -- --skip CLANGTIDY
> .venv/bin/spin quicklint -- --skip CLANGTIDY
> ```
>
> ```
> BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 USE_FLASH_ATTENTION=0 USE_MEM_EFF_ATTENTION=0 USE_DISTRIBUTED=1 USE_CUDA=1 USE_NCCL_EP=1 USE_SYSTEM_NCCL=1 .venv/bin/pip install -e . -v --no-build-isolation
> ```
>
> ```
> uv run --no-sync python test/distributed/test_c10d_nccl2.py ProcessGroupNCCL2ScalableInitTest.test_collective_with_scalable_init ProcessGroupNCCL2UnevenScalableInitTest.test_collective_with_scalable_init
> ```
>
> Authored with the assistance of an AI coding agent.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/191983
Approved by: https://github.com/kapilsh
ghstack dependencies: #191978, #191979
Sequence numbers always start at 0 on every rank now, so there is no
initial value for ranks to agree on, and `setSequenceNumberForGroup()`
has been an empty no-op for every backend (NCCL/Gloo/UCC) for a long
time. This removes the dead machinery while keeping the Backend-level
binding so downstream callers that use the backend form keep working.
`Backend::setSequenceNumberForGroup()` now emits a one-time deprecation
warning and does nothing. The `_set_sequence_number_for_group` binding is
kept on `Backend` because downstream projects call it on a backend object
(e.g. vLLM calls `backend_class._set_sequence_number_for_group()` on a
`ProcessGroupGloo`). The `ProcessGroup` method and its binding, the
redundant per-backend overrides, the `ProcessGroupWrapper`'s internal
call, and the internal callers in `init_process_group`/`split_group` are
removed. The `_get_sequence_number_for_group` path used for desync
detection is unchanged.
## History
The implementation has been a no-op for years. The original rank-0
`rand()` + store-broadcast implementation was added in #55319, then
stripped out per backend:
- NCCL: #68310 (`146a7f68e23`) replaced the body with an empty `{}`.
- Gloo: #101422 (`6f8a71aa3d7`) started gloo sequence numbers at 0.
This PR just removes the now-empty shells and the dead plumbing around
them.
## Downstream impact
Callers that invoke this on a **backend** object keep working: they hit
the retained `Backend` binding, which now warns once and no-ops. For
example vLLM calls `backend_class._set_sequence_number_for_group()` on a
`ProcessGroupGloo`.
- backend-form callers: https://github.com/search?q=%2Fbackend.*%5C._set_sequence_number_for_group%5C%28%5C%29%2F+-path%3Adistributed_c10d&type=code
Callers that invoke it on a **ProcessGroup** object are affected, since
this PR drops the `ProcessGroup` binding:
- pg-form callers: https://github.com/search?q=pg._set_sequence_number_for_group%28%29+-path%3Adistributed_c10d&type=code
## Test Plan
Build:
```
pip install -e . -v --no-build-isolation
```
Behavior check (Backend keeps a warning no-op, ProcessGroup binding
removed, getter intact, no exception):
```python
import os, warnings, torch, torch.distributed as dist
os.environ["MASTER_ADDR"] = "localhost"
os.environ["MASTER_PORT"] = "29577"
dist.init_process_group("gloo", rank=0, world_size=1)
pg = dist.group.WORLD
b = pg._get_backend(torch.device("cpu"))
assert not hasattr(pg, "_set_sequence_number_for_group")
assert hasattr(b, "_set_sequence_number_for_group")
b._set_sequence_number_for_group() # warns once, no-op, does not raise
assert b._get_sequence_number_for_group() == 0
dist.destroy_process_group()
```
Lint:
```
lintrunner -a
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/188611
Approved by: https://github.com/kapilsh
Summary:
Introduce the torchcomms `_single` collective names on the C++ `c10d::Backend` and migrate the in-tree backends to define them, while keeping the old names fully working for backward compatibility.
`Backend` now declares `all_gather_single`, `all_gather_single_coalesced`, `reduce_scatter_single`, `reduce_scatter_single_coalesced`, and `all_to_all_single` alongside the existing `_allgather_base`, `allgather_into_tensor_coalesced`, `_reduce_scatter_base`, `reduce_scatter_tensor_coalesced`, and `alltoall_base`, which are kept as overridable, forwarding aliases.
Backward compatibility is preserved in both directions: each new method and its old-name alias forward to each other, so a `Backend` subclass may override EITHER name and a caller may invoke EITHER name. The old aliases simply forward to the canonical `_single` method; each canonical method opens with the `C10D_BACKEND_FORWARDING_GUARD()` macro, which installs a function-local thread-local re-entry flag using `__func__` for the message. If a backend overrides neither name the mutual forwarding re-enters the canonical method, the flag trips, and it reports "does not support <method>" instead of recursing forever. As a result, existing callers of the old names (first-party and out-of-tree) and existing out-of-tree backends that override the old names both keep working unchanged. Removing the old overrides outright would have broken out-of-tree backends on the dispatch path, and dropping the old caller-facing names would have broken direct callers; the bidirectional forwarding avoids both.
The old names are intentionally NOT marked `C10_DEPRECATED_MESSAGE` in this change. PyTorch's open-source build compiles the bundled `third_party/torch-xpu-ops` submodule with `-Werror -Wdeprecated`, and its XPU collective op registration (`xccl/Register.cpp`) still calls the old `Backend` names, so a compile-time deprecation turns into a hard build error there. The deprecation will be reintroduced in a follow-up once those callers (and any other out-of-tree backends that are built in-tree) are migrated to the `_single` names and the submodule pin is bumped.
The in-tree backends are migrated to define the new names: `ProcessGroupNCCL`, `ProcessGroupGloo`, `ProcessGroupMPI`, `ProcessGroupUCC`, `ProcessGroupWrapper`, `FakeProcessGroup`, `NCCLXStub`, and the `torch_openreg` (`ProcessGroupOCCL`) test backend. The dispatcher op implementations in `Ops.cpp`, `ProcessGroupWrapper`'s forwarders, and the `Backend` pybind bindings in `init.cpp` use the new names.
The underlying aten / c10d dispatcher op names (`c10d::_allgather_base_`, `c10d::alltoall_base_`, etc.), their schema strings, the `IMPL_*` macro names in `Ops.cpp`, and the `OpType` enum values are intentionally left unchanged for backward compatibility.
Suggested review order: `Backend.hpp` (the new/old method pairs and the bidirectional forwarding + `ForwardingGuard`), then the per-backend subclass renames, then the `Ops.cpp` / `ProcessGroupWrapper` / `init.cpp` call sites.
This diff was authored with the assistance of an AI coding agent (Claude).
Test Plan:
This revision only removes the compile-time deprecation annotations (`C10_DEPRECATED_MESSAGE`), their `-Wdeprecated-declarations` suppressions, and the now-unused `<c10/util/Deprecated.h>` include. The `_single` rename and the bidirectional forwarding/recursion guard are unchanged, so there is no runtime or codegen change -- dropping the annotations only removes deprecation warnings.
Verified by open-source CI on the exported commit: the full libtorch build across platforms, the `linux-noble-xpu-n-py3.10 / build-osdc` job (which compiles the bundled `torch-xpu-ops` `xccl/Register.cpp` with `-Werror -Wdeprecated` and previously failed on `-Werror=deprecated-declarations`), and the `lintrunner-clang` format jobs.
Reviewed By: kapilsh
Differential Revision: D108364288
Pull Request resolved: https://github.com/pytorch/pytorch/pull/187140
Approved by: https://github.com/kapilsh
Summary:
Raise the compile-time minimum NCCL version to **2.23** and remove the preprocessor and runtime version gates that guarded features introduced in NCCL 2.23 or earlier. Gates for features introduced *after* 2.23 are deliberately preserved so the code still compiles and behaves correctly against NCCL 2.23 through 2.26.
The introduction version of every feature was cross-checked against the NCCL release notes and the public `nccl.h` headers (each gated symbol was confirmed present in the `v2.23.4` header where its gate is dropped, and absent there where its gate is retained) before deciding whether its gate could be removed:
| Feature | Introduced | Gate |
| --- | --- | --- |
| `ncclAvg` | 2.10 | removed |
| PreMulSum (`ncclRedOpCreatePreMulSum`) | 2.11 | removed |
| `ncclGetLastError` / `ncclRemoteError` | 2.13 | removed |
| non-blocking comms + `ncclConfig_t` | 2.14 | removed |
| `ncclCommSplit` | 2.18 | removed |
| `ncclCommRegister` / `ncclMemAlloc` | 2.19 | removed |
| group abort (`ncclGroupStart/End` for abort) | 2.22 | removed |
| `ncclCommInitRankScalable` | 2.23 | removed |
| FP8 `e5m2`/`e4m3` | 2.24 | kept |
| QoS / `trafficClass` | 2.26 | kept |
| `ncclCommShrink`, `ncclCommWindowRegister`, symmetric memory, `collnetEnable`/`CTAPolicy`/`nvlsCTAs` | 2.27 | kept |
## Suggested review order
1. **`NCCLUtils.hpp`** - the `static_assert` is bumped from 2.7 to 2.23. Macros for features at or below 2.23 are deleted and their feature code becomes unconditional (including `NCCL_HAS_INIT_RANK_SCALABLE`, which arrived in 2.23), while macros for features above 2.23 are retained (`NCCL_SUPPORTS_FP8` 2.24, `NCCL_HAS_QOS` 2.26, and the 2.27 `NCCL_HAS_COMM_WINDOW_REGISTER` / `NCCL_HAS_COLLNET` / `NCCL_HAS_CTA_POLICY` / `NCCL_HAS_NVLS_CTAS` / `NCCL_HAS_COMM_SHRINK`).
2. **`NCCLUtils.cpp`, `ProcessGroupNCCL.{hpp,cpp}`, `init.cpp`** - dead `#ifdef`/`#ifndef`/`#else` branches for <= 2.23 features are removed and the live branch inlined. The runtime `getNcclVersionNumber() >= 2.22` group-abort check is dropped (always true at the new minimum) and scalable init (2.23) becomes unconditional. The retained gates guard only post-2.23 features: the FP8 dtype mapping (2.24), the QoS / collnet / CTA-policy / NVLS config fields (2.26-2.27), and `ncclCommShrink` / window registration (2.27).
3. **`cuda/nccl.{h,cpp}`** - the ancient NCCL 1.x / 2.0 / 2.7 / 2.12.10 / 2.13 / 2.14 guards are removed; only NCCL-presence gates (`USE_NCCL`, `defined(NCCL_MAJOR)`) and `>= 2.28` gates remain.
4. **`symm_mem/nccl_dev_cap.hpp`** - `NCCL_HAS_SYMMEM_SUPPORT` stays a normal `>= 2.27` version gate, since symmetric memory is not available at the new 2.23 minimum.
## Kept intentionally
Every gate for a feature newer than 2.23 is untouched: `NCCL_SUPPORTS_FP8` (2.24), `NCCL_HAS_QOS` (2.26), the 2.27 symmetric-memory / shrink / window-registration / collnet / CTA-policy / NVLS-CTAs gates, the pre-existing `>= 2.28` gates (`ncclAlltoAll`, `NCCL_HAS_SYMMEM_DEVICE_SUPPORT`, `NCCL_HAS_COMM_OFFLOAD` 2.29.7, `NCCL_HAS_MAX_P2P_PEERS` 2.30, etc.), and the defensive runtime `version >= 2.27` check inside `ProcessGroupNCCL::shrink()`.
`HAS_NCCL_BF16_DATATYPE` (`nccl.h`) keeps its CUDA-bf16 / ROCm structure with the simplified `NCCL_MAJOR >= 2` subexpression (the bf16 dtype arrived in 2.10, below the new minimum), preserving the "NCCL absent => disabled" behavior its consumers (e.g. `quantization_gpu.cu`) rely on.
Test Plan:
This change is preprocessor-only; the new `static_assert` enforces the NCCL >= 2.23 minimum and the retained gates keep the 2.24-2.27 features behind their version checks.
Linted every changed file (fast linters, no issues):
```
arc lint fbcode/caffe2/torch/csrc/distributed/c10d/NCCLUtils.cpp fbcode/caffe2/torch/csrc/distributed/c10d/NCCLUtils.hpp fbcode/caffe2/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp fbcode/caffe2/torch/csrc/distributed/c10d/ProcessGroupNCCL.hpp fbcode/caffe2/torch/csrc/distributed/c10d/init.cpp fbcode/caffe2/torch/csrc/distributed/c10d/symm_mem/nccl_dev_cap.hpp fbcode/caffe2/torch/csrc/cuda/nccl.h
```
Compilation against the full range of supported NCCL versions (2.23 through current) is exercised by CI; build against any NCCL >= 2.23, e.g.:
```
uv pip install -e . --no-build-isolation -v
```
---
This PR was authored with the assistance of Claude, an AI coding assistant.
Reviewed By: kapilsh
Differential Revision: D107591575
Pulled By: d4l3k
Pull Request resolved: https://github.com/pytorch/pytorch/pull/186292
Approved by: https://github.com/kapilsh
## Summary
Raise the compile-time minimum NCCL version to **2.27** and remove the preprocessor and runtime version gates that guarded features introduced in NCCL 2.27 or earlier.
The introduction version of every gated feature was cross-checked against the NCCL release notes before its gate was removed:
| Feature | Introduced |
| --- | --- |
| `ncclAvg` | 2.10 |
| PreMulSum (`ncclRedOpCreatePreMulSum`) | 2.11 |
| `ncclGetLastError` / `ncclRemoteError` | 2.13 |
| non-blocking comms + `ncclConfig_t` | 2.14 |
| `ncclCommSplit` | 2.18 |
| `ncclCommRegister` / `ncclMemAlloc` | 2.19 |
| group abort (`ncclGroupStart/End` for abort) | 2.22 |
| `ncclCommInitRankScalable` | 2.23 |
| FP8 `e5m2`/`e4m3` | 2.24 |
| QoS / `trafficClass` | 2.26 |
| `ncclCommShrink`, `ncclCommWindowRegister`, symmetric memory, `collnetEnable`/`CTAPolicy`/`nvlsCTAs` | 2.27 |
## Suggested review order
1. **`NCCLUtils.hpp`** — the `static_assert` is bumped from 2.7 to 2.27 and the now-unconditional feature macros are deleted. This makes every gate below dead code.
2. **`NCCLUtils.cpp`, `ProcessGroupNCCL.{hpp,cpp}`, `init.cpp`** — dead `#ifdef`/`#ifndef`/`#else` branches are removed and the live branch inlined; the runtime `getNcclVersionNumber() >= 2.22` (group abort) and `version < 2.19` (mem allocator) checks are dropped.
3. **`cuda/nccl.{h,cpp}`** — the local 2.13/2.14 feature macros and the ancient NCCL 1.x / 2.0 / 2.7 / 2.12.10 guards are removed; only NCCL-presence gates (`USE_NCCL`, `defined(NCCL_MAJOR)`) and `>= 2.28` gates remain.
4. **`symm_mem/nccl_dev_cap.hpp`** and the C++ test — `NCCL_HAS_SYMMEM_SUPPORT` is made unconditional within `USE_NCCL`.
## Kept intentionally
Gates for versions newer than 2.27 are untouched (`>= 2.28` `ncclAlltoAll`, `NCCL_HAS_SYMMEM_DEVICE_SUPPORT`, `NCCL_HAS_COMM_OFFLOAD` 2.29.7, `NCCL_HAS_MAX_P2P_PEERS` 2.30, etc.), as is the defensive runtime `version >= 2.27` check inside `ProcessGroupNCCL::shrink()`.
Two macros were not deleted outright because they encode NCCL **availability**, not just a version, and are evaluated in non-NCCL build configs:
- `HAS_NCCL_BF16_DATATYPE` (`nccl.h`) keeps its CUDA-bf16/ROCm structure; only the `NCCL_MINOR >= 10` subexpression is dropped (now `NCCL_MAJOR >= 2`), preserving the "NCCL absent => disabled" behavior its consumers (e.g. `quantization_gpu.cu`) rely on.
- `NCCL_HAS_SYMMEM_SUPPORT` is kept as a macro but defined unconditionally inside `#if USE_NCCL`, since the symmetric-memory source files use it to gate on NCCL presence as well as version.
## Test plan
Build against NCCL 2.29.7 (satisfies the new `static_assert`):
```
uv pip install -e . --no-build-isolation -v
```
Lint the changed files:
```
lintrunner -a <changed files>
```
---
This PR was authored with the assistance of Claude, an AI coding assistant.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/186163
Approved by: https://github.com/ngimel, https://github.com/kapilsh, https://github.com/eqy
Reported in https://github.com/pytorch/pytorch/issues/143470, we have a dangling references in `CudaEventCache`. So we want to fix it.
1. We add a unit test to repro the issue mentioned in the issue.
2. Instead of converting variables to shared pointers as suggested in the issue, we then make the cache itself a shared pointer. So if the thread creates the cache dies before all events get recycled, the cache is still there until the last CudaEvent get deleted. (thanks for the suggestion from @kwen2501 )
Pull Request resolved: https://github.com/pytorch/pytorch/pull/144496
Approved by: https://github.com/kwen2501
We added `CudaEventCache` in https://github.com/pytorch/pytorch/pull/133727 and this is a feature which tries to reuse CudaEvent so that we don't call destroy of CudaEvent which causes hang in the past. We had a bunch of tests and testing on TorchTitan and internal workload already. So far no errors or crash are found at the moment so we decide to roll out to all OSS users. For internal workload, this PR would not affect it because of some internal gating.
Also we observed some multi-device use cases in OSS, so that we want to bring back multi-device support originally proposed in https://github.com/pytorch/pytorch/pull/122732/files.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/140975
Approved by: https://github.com/eqy, https://github.com/kwen2501
Previously:
```
[W Utils.hpp:133] Warning: Environment variable NCCL_ASYNC_ERROR_HANDLING is deprecated; use TORCH_NCCL_ASYNC_ERROR_HANDLING instead (function getCvarInt)
[W Utils.hpp:133] Warning: Environment variable NCCL_ASYNC_ERROR_HANDLING is deprecated; use TORCH_NCCL_ASYNC_ERROR_HANDLING instead (function getCvarInt)
```
With this PR, those warnings disappear. They were introduced in #114077
This change was generated with this sed script, applied with `sed -i -f /tmp/x **/*.{py,hpp,cpp,cc}` and hand inspected.
```
s/\bNCCL_BLOCKING_WAIT\b/TORCH_NCCL_BLOCKING_WAIT/g
s/\bNCCL_ENABLE_TIMING\b/TORCH_NCCL_ENABLE_TIMING/g
s/\bNCCL_DESYNC_DEBUG\b/TORCH_NCCL_DESYNC_DEBUG/g
s/\bNCCL_ASYNC_ERROR_HANDLING\b/TORCH_NCCL_ASYNC_ERROR_HANDLING/g
s/\bENABLE_NCCL_HEALTH_CHECK\b/TORCH_ENABLE_NCCL_HEALTH_CHECK/g
s/\bNCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK\b/TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK/g
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/114880
Approved by: https://github.com/kwen2501
- [c10d] (retry) Opportunistically use `ncclCommSplit` when creating new NCCL groups (#112889)
- Guard use of `split_from` with a `hasattr` check for cases when NCCL (or RCCL) lacks `ncclCommSplit`
Fixes cause of revert of original PR
Pull Request resolved: https://github.com/pytorch/pytorch/pull/114385
Approved by: https://github.com/huydhn
Currently `ncclCommInitRankConfig` is always used when creating new
communicator groups. This is wasteful as it creates non-shared pairs
of endpoint queues as well as costs time to re-establish
communication.
This change is transparent and opportunistic; when `dist.new_group` is
called, it will use the existing, healthy world process group to
select the right ranks to include in the process group.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/112889
Approved by: https://github.com/kwen2501
NCCL_ prefix should only be used for NCCL library's environment variables. We currently use a few environment variables in PyTorch with the NCCL_ prefix that are the NCCL library does not understand.
This patch renames such environment variables to use the TORCH_NCCL_ prefix instead. We still maintain the old NCCL_ variables, but throw a warning when they are used.
The following env changes have been made:
`NCCL_BLOCKING_WAIT` -> `TORCH_NCCL_BLOCKING_WAIT`
`NCCL_ENABLE_TIMING` -> `TORCH_NCCL_ENABLE_TIMING`
`NCCL_DESYNC_DEBUG` -> `TORCH_NCCL_DESYNC_DEBUG`
`NCCL_ASYNC_ERROR_HANDLING` -> `TORCH_NCCL_ASYNC_ERROR_HANDLING`
`ENABLE_NCCL_HEALTH_CHECK` -> `TORCH_ENABLE_NCCL_HEALTH_CHECK`
`NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK` -> `TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK`
Fixes #ISSUE_NUMBER
Pull Request resolved: https://github.com/pytorch/pytorch/pull/114077
Approved by: https://github.com/fduwjj
Summary:
The getCvar* functions allow us to provide multiple environment variables for the same value. This allows us to deprecate some variables in favor of others, while still allowing users to temporarily use the old variables for some time.
Test Plan: OSS CI
Reviewed By: fduwjj, XilunWu
Differential Revision: D51225487
Fixes #ISSUE_NUMBER
Pull Request resolved: https://github.com/pytorch/pytorch/pull/113797
Approved by: https://github.com/fduwjj
If code is compiled without `glog`, there are no way to control log levels other than explicitly calling `c10::initLogging()`
Test plan: Run `TORCH_CPP_LOG_LEVEL=0 ./bin/ProcessGroupNCCLTest` and observe extra log messages
Pull Request resolved: https://github.com/pytorch/pytorch/pull/112809
Approved by: https://github.com/fduwjj
Avoid changing default for other backends as CPU backend (GLOO) may need
longer timeouts.
Motivated by trying to save cluster time when encountering collective
hangs. Generally collectives should time out within seconds and 30
minutes (or 10 minutes) should provide ample headroom for edge cases.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/110947
Approved by: https://github.com/xw285cornell, https://github.com/fduwjj
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/88330
### Implementation
Move backend-specific (NCCL, Gloo, etc) collective implementations to corresponding `Backend` class. Update ProcessGroup to support multiple backends and use dispatcher to calls backends based on tensor device type.
### Changes
#### c++ changes (ProcessGroup files, `Ops.cpp`, `init.cpp`)
- Update pybind definitions for new process group base class and new backend class
- Update pybinded backend class with collective definitions to keep BC with Python PG instances (e.g. `dist.ProcessGroupGloo`, `dist.ProcessGroupNCCL`) which are used in tests
- Switch `ProcessGroupGloo`, `ProcessGroupNCCL`, `ProcessGroupMPI`, `ProcessGroupUCC` to derive from the `Backend` class.
- Update CPU/CUDA `Ops.cpp` and `OpsImpl.cpp` to perform this dispatching by querying the backend using the device type
- Update internal dispatched implementation of `barrier` to use a tensor which allows operation to be dispatched.
- Update `allgather` collective to use `TensorList`. For some reason it was using the default implementation of `allgather` rather than dispatching it correctly. I still don't understand why and had originally filed an issue in 85122.
#### python changes (`distributed_c10d.py`, test files)
- Add BackendConfig class to specify the default configurations of backends and `get_backend_config()` API
- `get_backend()` deprecation warning
- `init_process_group` how returns a generic `ProcessGroup` object, it contains a list of backends (the ones stated above) which it will dispatch operations to.
- `new_group` updated to return the same as above
- Update `test_c10d_gloo.py`, Update `DistributedDataParallelTest` to use `init_process_group`, Update `ReducerTest`, update `test_broadcast_coalesced_gloo` to move from PG instance and gloo options
- Update `test_c10d_nccl.py`, Update `DistributedDataParallelTest` to use `init_process_group`
- Specific tests updated: `test_Backend_enum_class`
### Changes missing
- lazy initialization of backends
- support parsing of BackendConfig
### open questions
- Pure Python PG extensions (https://github.com/pytorch/pytorch/pull/66338)
# Example
This is a basic script (using 2 backends within a process group)
```python
# python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 basic_scenario.py
import torch.distributed as dist
import torch
import os
if __name__ == "__main__":
rank = os.environ.get("RANK")
# initialize with both gloo and nccl
dist.init_process_group()
# with gloo
dist.all_reduce(torch.tensor([1.0]))
print(f"Rank {rank} finished")
# with nccl
dist.all_reduce(torch.tensor([1.0], device=f"cuda:{rank}"))
```
Test Plan: Imported from OSS
Differential Revision: D42069829
Pulled By: H-Huang
Pull Request resolved: https://github.com/pytorch/pytorch/pull/90997
Approved by: https://github.com/awgu, https://github.com/fduwjj
Headers under torch/csrc/distributed may be referened with relative path, e.g., "<c10d/...>". However, relative path cannot be gracefully handled by Meta internal build when the NCCL PG is hipified to support AMD/RCCL because the "hipified" header files are generated in other directories. Moreover, using absolute path for header inclusion is the state-of-the-art in most components in Pytorch. Thus, this patch refactors all header paths in torch/csrc/distributed to be absolute.
See D39835774 for more details about Meta internal complication.
**How to test**: commit 9e5d199 removes -I./torch/csrc/distributed in compile options. Thus use it to verify we don't miss any relative path use of torch/csrc/distributed headers.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85780
Approved by: https://github.com/kumpera, https://github.com/huydhn
Headers under torch/csrc/distributed may be referened with relative path, e.g., "<c10d/...>". However, relative path cannot be gracefully handled by Meta internal build when the NCCL PG is hipified to support AMD/RCCL because the "hipified" header files are generated in other directories. Moreover, using absolute path for header inclusion is the state-of-the-art in most components in Pytorch. Thus, this patch refactors all header paths in torch/csrc/distributed to be absolute.
See D39835774 for more details about Meta internal complication.
**How to test**: commit 9e5d199 removes -I./torch/csrc/distributed in compile options. Thus use it to verify we don't miss any relative path use of torch/csrc/distributed headers.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85780
Approved by: https://github.com/kumpera
### Changes
- Move ProcessGroup::Work into its own class and update all the references to it / header includes.
#### Motivation
In the future PRs we will repurpose ProcessGroup to instead contain a list of Backends (ProcessGroupNCCL/Gloo/UCC) and perform dispatching to them based on tensor type. This change is prevent a circular dependency with ProcessGroup depending on Backend and Backend depending on ProcessGroup::Work.
Differential Revision: [D38839212](https://our.internmc.facebook.com/intern/diff/D38839212)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/83680
Approved by: https://github.com/kwen2501
Tristan Rice