mirror of
https://github.com/pytorch/pytorch.git
synced 2026-10-28 18:56:51 +08:00
Tristan Rice
2f2f675bc5
ProcessGroupNCCLErrorsTest spends ~830s per CUDA distributed config in three tests:
- testNCCLErrorsNonBlocking (242s): the watchdog sleeps TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC * 4 (60s by default) after each of the 4 failed works. Set it to 100ms, as the other timeout tests already do.
- testTimeoutTearsDownWithoutRethrow{,WithCleanUp} (~290s each): the death-test child aborts with a CUDA context, and CI leaves core dumps enabled. A core of a CUDA process is ~24GB sparse / ~2GB resident. Disable core dumps in the child with RLIMIT_CORE=0. The death assertion is unchanged.
This PR was written with AI assistance (Claude).
Test plan:
`build/bin/ProcessGroupNCCLErrorsTest --gtest_filter='*NonBlocking:*TearsDown*'` locally on 1 GPU: NonBlocking 241644ms -> 1848ms; TearsDown tests pass in 1.5s and 2.0s. The local core_pattern discards cores, so the TearsDown saving can only be confirmed by CI durations (trunk: 290s / 287s, run 36203003955).
Pull Request resolved: https://github.com/pytorch/pytorch/pull/199826
Approved by: https://github.com/pytorchgreenlight