Fixes#106700.
`torch::nn::functional::multi_head_attention_forward` divides before it validates:
```cpp
const auto head_dim = embed_dim / num_heads;
TORCH_CHECK(head_dim * num_heads == embed_dim, "embed_dim must be divisible by num_heads");
```
so `num_heads == 0` is an integer division by zero. That is not a catchable exception — the
process dies. Running the call in a subprocess on Windows exits with `0xC0000094`
(`STATUS_INTEGER_DIVIDE_BY_ZERO`) with no stdout, no stderr and no traceback; the original report
shows the same as SIGFPE on Linux.
### Half of this issue is already fixed — this is the other half
#186376 added `TORCH_CHECK(options.num_heads() > 0, ...)` to `MultiheadAttentionImpl::reset`
(`activation.cpp:504`) and landed as `e71ac681afa`. That closed the case in the issue *body*: the
module constructor. The reporter raised a second case in a follow-up comment — the functional
entry point — and it was never covered. This adds the same check with the same message,
immediately before the division.
@malfet approved the earlier #106778, which fixed both halves; it went stale on unrelated CI in
2023 and closed without landing.
### Test
`ModulesTest.MultiheadAttentionZeroNumHeads` covers both halves — the module constructor and the
functional call. #186376 already covered the constructor with `MultiheadAttentionRejectsZeroNumHeads`;
this adds the missing coverage for the functional entry point.
The crash cannot be caught in-process, so I verified through a `cpp_extension` harness run in a
subprocess. Against current headers:
```
stdout: (empty)
stderr: (empty)
exit code: 3221225620 # 0xC0000094, STATUS_INTEGER_DIVIDE_BY_ZERO
```
With this change, the same harness:
```
stdout: RAISED RuntimeError num_heads must be greater than 0
exit code: 0
```
`cl.exe /Zs test\cpp\api\modules.cpp` is clean against the build's flags; the gtest binary itself
is left to CI, as this machine cannot launch freshly linked executables.
Developed with AI assistance (Claude); I reproduced the crash, wrote the fix and ran the checks
above.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/195381
Approved by: https://github.com/drisspg
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Fixes#149120
Adds a `keepdim` parameter to `torch.cosine_similarity`, consistent with other reduction-like ops (`sum`, `norm`, `pairwise_distance`).
```python
>>> x1 = torch.randn(2, 3, 4)
>>> x2 = torch.randn(2, 3, 4)
>>> torch.cosine_similarity(x1, x2, dim=1).shape
torch.Size([2, 4])
>>> torch.cosine_similarity(x1, x2, dim=1, keepdim=True).shape
torch.Size([2, 1, 4])
```
**Changes:**
- `native_functions.yaml`: Add `bool keepdim=False` to `cosine_similarity` schema
- `Distance.cpp`: Pass `keepdim` to final `.sum(dim, keepdim)` call
- C++ API: Add `keepdim` to `CosineSimilarityOptions`, plumb through functional and module layers
- ONNX symbolic: Accept `keepdim` and pass to `keepdims_i` in reduce ops
- `torch/overrides.py`: Add `keepdim` to override lambda
- Tests: Add `keepdim=True` shape and value tests in Python (`test_nn.py`, OpInfo samples) and C++ (`modules.cpp`)
The default `keepdim=False` preserves full backward compatibility.
**Note:** Per the FC window policy, a follow-up PR will update the Python frontend (`F.cosine_similarity` docstring and `nn.CosineSimilarity` module) to expose `keepdim` after these C++ changes land.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/189654
Approved by: https://github.com/albanD
Co-authored-by: albanD via mergedog <desmaison.alban@gmail.com>
# Fix: Division-by-Zero in `MultiheadAttentionImpl::reset()` When `num_heads == 0`
## Summary
Fixes#186237
Fixed a division-by-zero in `MultiheadAttentionImpl::reset()` triggered when `num_heads` is `0`.
Previously, `head_dim` was computed before any validation of `num_heads`:
```cpp
head_dim = options.embed_dim() / options.num_heads();
```
Constructing `MultiheadAttention` with `num_heads = 0` could invoke undefined behavior (integer division by zero / floating-point exception) before any validation logic was reached.
This PR adds a `TORCH_CHECK` to validate that `num_heads > 0` **before** performing the division, replacing the silent UB with a clear, actionable error message.
## Changes
- Added a `TORCH_CHECK(options.num_heads() > 0, ...)` guard in `MultiheadAttentionImpl::reset()` prior to the `head_dim` computation.
## Test Plan
Added a C++ API test verifying that constructing:
```cpp
MultiheadAttention(MultiheadAttentionOptions(0, 0))
```
throws a `c10::Error`.
**Before this change:** the test triggered a floating-point exception due to division by zero.
**After this change:** the test passes and the invalid configuration is reported via a proper error.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/186376
Approved by: https://github.com/drisspg
BE project hard-removing the deprecated named tensor feature to remove overhead and code bloat. This was nearly all done by wrestling with Claude Code, so it needs some personal review and probably needs to be split into multiple PRs to help reviewers.
The process was me absent-mindedly doing this throughout the week while working on other stuff:
```
me: remove named tensor along with all associated APIs, tests, and docs from the codebase
claude: <does stuff>
while not built:
me: <tries to build>
if error:
me: there are still remnants of named tensor (with build-error related example); remove all these
claude: <does more stuff, usually very good at generalizing from the example>
```
This wasn't particularly efficient (it built after ~27 iterations) but didn't require much thought or time from my end. The only part Claude Code did NOT do is add a couple now-necessary `#include <ATen/WrapDimUtils.h>` statements.
It builds! Testing is still a WIP.
Having Claude Code break this into manageable PRs, the breakdown is:
```
1. Remove named tensor tests and documentation
- test/test_namedtensor.py
- test/compiled_autograd_skips/TestNamedTensor.*
- test/dynamo_expected_failures/TestNamedTensor.*
- docs/source/named_tensor.md
- docs/source/name_inference.md
- Test file updates (test_ops.py, test_overrides.py, test_builtins.py, test_torch.py, etc.)
- functorch/op_analysis/annotated_ops
- Expect file updates (HasDecompTest)
- test/allowlist_for_publicAPI.json
2. Remove named tensor operators from native_functions.yaml
- All Dimname-based operator overloads
- align_tensors, align_as, align_to, rename, refine_names
- Factory function names variants (empty.names, zeros.names, etc.)
3. Remove Python named tensor APIs
- torch/_namedtensor_internals.py
- torch/_tensor.py (names property, rename methods)
- torch/functional.py (align_tensors)
- torch/overrides.py references
- torch/_dynamo/trace_rules.py references
- torch/multiprocessing/reductions.py references
4. Remove C++ named tensor infrastructure and build system
- Dimname type system (Dimname.h/cpp, python_dimname.h/cpp, python_arg_parser changes)
- Core infrastructure (NamedTensor.h/cpp, NamedTensorUtils.h/cpp, TensorNames.h/cpp)
- TensorImpl changes (NamedTensorMetaInterface)
- Structured kernels (names parameter from set_output_* and all call sites)
- Guards (NoNamesGuard, NamesMode, guard.reset() calls)
- Build system (CMakeLists.txt, build_variables.bzl, cmake/prioritized_text.txt)
- Codegen (torchgen/, tools/autograd/ templates)
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/173895
Approved by: https://github.com/malfet, https://github.com/guangyey, https://github.com/cota
BE project hard-removing the deprecated named tensor feature to remove overhead and code bloat. This was nearly all done by wrestling with Claude Code, so it needs some personal review and probably needs to be split into multiple PRs to help reviewers.
The process was me absent-mindedly doing this throughout the week while working on other stuff:
```
me: remove named tensor along with all associated APIs, tests, and docs from the codebase
claude: <does stuff>
while not built:
me: <tries to build>
if error:
me: there are still remnants of named tensor (with build-error related example); remove all these
claude: <does more stuff, usually very good at generalizing from the example>
```
This wasn't particularly efficient (it built after ~27 iterations) but didn't require much thought or time from my end. The only part Claude Code did NOT do is add a couple now-necessary `#include <ATen/WrapDimUtils.h>` statements.
It builds! Testing is still a WIP.
Having Claude Code break this into manageable PRs, the breakdown is:
```
1. Remove named tensor tests and documentation
- test/test_namedtensor.py
- test/compiled_autograd_skips/TestNamedTensor.*
- test/dynamo_expected_failures/TestNamedTensor.*
- docs/source/named_tensor.md
- docs/source/name_inference.md
- Test file updates (test_ops.py, test_overrides.py, test_builtins.py, test_torch.py, etc.)
- functorch/op_analysis/annotated_ops
- Expect file updates (HasDecompTest)
- test/allowlist_for_publicAPI.json
2. Remove named tensor operators from native_functions.yaml
- All Dimname-based operator overloads
- align_tensors, align_as, align_to, rename, refine_names
- Factory function names variants (empty.names, zeros.names, etc.)
3. Remove Python named tensor APIs
- torch/_namedtensor_internals.py
- torch/_tensor.py (names property, rename methods)
- torch/functional.py (align_tensors)
- torch/overrides.py references
- torch/_dynamo/trace_rules.py references
- torch/multiprocessing/reductions.py references
4. Remove C++ named tensor infrastructure and build system
- Dimname type system (Dimname.h/cpp, python_dimname.h/cpp, python_arg_parser changes)
- Core infrastructure (NamedTensor.h/cpp, NamedTensorUtils.h/cpp, TensorNames.h/cpp)
- TensorImpl changes (NamedTensorMetaInterface)
- Structured kernels (names parameter from set_output_* and all call sites)
- Guards (NoNamesGuard, NamesMode, guard.reset() calls)
- Build system (CMakeLists.txt, build_variables.bzl, cmake/prioritized_text.txt)
- Codegen (torchgen/, tools/autograd/ templates)
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/173895
Approved by: https://github.com/malfet, https://github.com/guangyey
BE project hard-removing the deprecated named tensor feature to remove overhead and code bloat. This was nearly all done by wrestling with Claude Code, so it needs some personal review and probably needs to be split into multiple PRs to help reviewers.
The process was me absent-mindedly doing this throughout the week while working on other stuff:
```
me: remove named tensor along with all associated APIs, tests, and docs from the codebase
claude: <does stuff>
while not built:
me: <tries to build>
if error:
me: there are still remnants of named tensor (with build-error related example); remove all these
claude: <does more stuff, usually very good at generalizing from the example>
```
This wasn't particularly efficient (it built after ~27 iterations) but didn't require much thought or time from my end. The only part Claude Code did NOT do is add a couple now-necessary `#include <ATen/WrapDimUtils.h>` statements.
It builds! Testing is still a WIP.
Having Claude Code break this into manageable PRs, the breakdown is:
```
1. Remove named tensor tests and documentation
- test/test_namedtensor.py
- test/compiled_autograd_skips/TestNamedTensor.*
- test/dynamo_expected_failures/TestNamedTensor.*
- docs/source/named_tensor.md
- docs/source/name_inference.md
- Test file updates (test_ops.py, test_overrides.py, test_builtins.py, test_torch.py, etc.)
- functorch/op_analysis/annotated_ops
- Expect file updates (HasDecompTest)
- test/allowlist_for_publicAPI.json
2. Remove named tensor operators from native_functions.yaml
- All Dimname-based operator overloads
- align_tensors, align_as, align_to, rename, refine_names
- Factory function names variants (empty.names, zeros.names, etc.)
3. Remove Python named tensor APIs
- torch/_namedtensor_internals.py
- torch/_tensor.py (names property, rename methods)
- torch/functional.py (align_tensors)
- torch/overrides.py references
- torch/_dynamo/trace_rules.py references
- torch/multiprocessing/reductions.py references
4. Remove C++ named tensor infrastructure and build system
- Dimname type system (Dimname.h/cpp, python_dimname.h/cpp, python_arg_parser changes)
- Core infrastructure (NamedTensor.h/cpp, NamedTensorUtils.h/cpp, TensorNames.h/cpp)
- TensorImpl changes (NamedTensorMetaInterface)
- Structured kernels (names parameter from set_output_* and all call sites)
- Guards (NoNamesGuard, NamesMode, guard.reset() calls)
- Build system (CMakeLists.txt, build_variables.bzl, cmake/prioritized_text.txt)
- Codegen (torchgen/, tools/autograd/ templates)
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/173895
Approved by: https://github.com/malfet, https://github.com/guangyey
Similar to #140425, we are making the implementation usable via header-only code sharing.
Review note: #62546 by @yanbing-j removed expm1 usage from this path. I don't know why and expm1 should be more efficient, so I've put it back. Please let me know if there is a good reason I shouldn't.
Testing: existing correctness tests should cover.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/149673
Approved by: https://github.com/cyyever, https://github.com/Skylion007
Fixes#95796
### Implementation
Adds python implementation for `nn.ZeroPad1d` and `nn.ZeroPad3d` in `torch/nn/modules/padding.py`.
Adds cpp implementation for `nn::ZeroPad1d` and `nn::ZeroPad3d` in the following 3 files, refactored with templates similarly to `nn::ConstantPad`'s implementation: <br>
- `torch/crsc/api/include/torch/nn/modules/padding.h`
- `torch/csrc/api/include/torch/nn/options/padding.h`
- `torch/csrc/api/src/nn/modules/padding.cpp`
Also added relevant definitions in `torch/nn/modules/__init__.py`.
### Testing
Adds the following tests:
- cpp tests of similar length and structure as `ConstantPad` and the existing `ZeroPad2d` impl in `test/cpp/api/modules.cpp`
- cpp API parity tests in `torch/testing/_internal/common_nn.py`
- module init tests in `test/test_module_init.py`
Also added relevant definitions in `test/cpp_api_parity/parity-tracker.md`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/96295
Approved by: https://github.com/soulitzer
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/69394
Modified loops in files under fbsource/fbcode/caffe2/ from the format
```
for(TYPE var=x0;var<x_max;x++)
```
to the format
```
for(const auto var: irange(xmax))
```
This was achieved by running r-barnes's loop upgrader script (D28874212) with some modification to exclude all files under /torch/jit and a number of reversions or unused variable suppression warnings added by hand.
Test Plan: Sandcastle
Reviewed By: malfet
Differential Revision: D32837991
fbshipit-source-id: fc7c4f76d2f32a17a0faf329294b3fe7cb81df32
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/66743
Modified loops in files under fbsource/fbcode/caffe2/ from the format
`for(TYPE var=x0;var<x_max;x++)`
to the format
`for(const auto var: irange(xmax))`
This was achieved by running r-barnes's loop upgrader script (D28874212) with some modification to exclude all files under /torch/jit and a number of reversions or unused variable suppression warnings added by hand.
Test Plan: Sandcastle
Reviewed By: malfet
Differential Revision: D31705359
fbshipit-source-id: c9ea2fbc0f9cd29e97a52dcb203addc5f2abb09b
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/66234
Modified loops in files under fbsource/fbcode/caffe2/ from the format
`for(TYPE var=x0;var<x_max;x++)`
to the format
`for(const auto var: irange(xmax))`
This was achieved by running r-barnes's loop upgrader script (D28874212) with some modification to exclude all files under /torch/jit and a number of reversions or unused variable suppression warnings added by hand.
bypass_size_limit
allow-large-files
Test Plan: Sandcastle
Reviewed By: ngimel
Differential Revision: D30652629
fbshipit-source-id: 0ae6c4bbbb554bad42e372792a6430e1acf15e3e
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/63612
This makes Tensor inherit from a new class TensorBase, that provides a subset of Tensor that doesn't
directly depend on native_functions.yaml. Code that only includes TensorBase.h with thus not need to
be rebuilt every time someone changes an operator signature.
Making `Tensor` inherit from this class means that `const TensorBase&` parameters will be callable
with an ordinary `Tensor`. I've also made `Tensor` constructible and assignable from `TensorBase` to
minimize friction in code mixing the two types.
To help enforce that `Tensor.h` and `Functions.h` aren't accidentally included, I've added an error
into `Operators.h` if `TORCH_ASSERT_NO_OPERATORS` is defined. We can either set this in the build
system for certain folders, or just define it at the top of any file.
I've also included an example of manually special-casing the commonly used `contiguous` operator.
The inline function's slow path defers to `TensorBase::__dispatch_contiguous` which is defined in
`Tensor.cpp`. I've made it so `OptionalTensorRef` is constructible from `TensorBase`, so I can
materialize a `Tensor` for use in dispatch without actually increasing its refcount.
Test Plan: Imported from OSS
Reviewed By: gchanan
Differential Revision: D30728580
Pulled By: ezyang
fbshipit-source-id: 2cbc8eee08043382ee6904ea8e743b1286921c03
Summary:
Enable Gelu bf16/fp32 in CPU path using Mkldnn implementation. User doesn't need to_mkldnn() explicitly. New Gelu fp32 performs better than original one.
Add Gelu backward for https://github.com/pytorch/pytorch/pull/53615.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/58525
Reviewed By: ejguan
Differential Revision: D29940369
Pulled By: ezyang
fbshipit-source-id: df9598262ec50e5d7f6e96490562aa1b116948bf
Summary:
As GoogleTest `TEST` macro is non-compliant with it as well as `DEFINE_DISPATCH`
All changes but the ones to `.clang-tidy` are generated using following script:
```
for i in `find . -type f -iname "*.c*" -or -iname "*.h"|xargs grep cppcoreguidelines-avoid-non-const-global-variables|cut -f1 -d:|sort|uniq`; do sed -i "/\/\/ NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)/d" $i; done
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/62008
Reviewed By: driazati, r-barnes
Differential Revision: D29838584
Pulled By: malfet
fbshipit-source-id: 1b2f8602c945bd4ce50a9bfdd204755556e31d13
Summary:
Fixes https://github.com/pytorch/pytorch/issues/27655
This PR adds a C++ and Python version of ReflectionPad3d with structured kernels. The implementation uses lambdas extensively to better share code from the backward and forward pass.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/59791
Reviewed By: gchanan
Differential Revision: D29242015
Pulled By: jbschlosser
fbshipit-source-id: 18e692d3b49b74082be09f373fc95fb7891e1b56
Summary:
This is an automatic change generated by the following script:
```
#!/usr/bin/env python3
from subprocess import check_output, check_call
import os
def get_compiled_files_list():
import json
with open("build/compile_commands.json") as f:
data = json.load(f)
files = [os.path.relpath(node['file']) for node in data]
for idx, fname in enumerate(files):
if fname.startswith('build/') and fname.endswith('.DEFAULT.cpp'):
files[idx] = fname[len('build/'):-len('.DEFAULT.cpp')]
return files
def run_clang_tidy(fname):
check_call(["python3", "tools/clang_tidy.py", "-c", "build", "-x", fname,"-s"])
changes = check_output(["git", "ls-files", "-m"])
if len(changes) == 0:
return
check_call(["git", "commit","--all", "-m", f"NOLINT stubs for {fname}"])
def main():
git_files = check_output(["git", "ls-files"]).decode("ascii").split("\n")
compiled_files = get_compiled_files_list()
for idx, fname in enumerate(git_files):
if fname not in compiled_files:
continue
if fname.startswith("caffe2/contrib/aten/"):
continue
print(f"[{idx}/{len(git_files)}] Processing {fname}")
run_clang_tidy(fname)
if __name__ == "__main__":
main()
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56892
Reviewed By: H-Huang
Differential Revision: D27991944
Pulled By: malfet
fbshipit-source-id: 5415e1eb2c1b34319a4f03024bfaa087007d7179
Summary:
This PR adds a `padding_idx` parameter to `nn.EmbeddingBag` and `nn.functional.embedding_bag`. As with `nn.Embedding`'s `padding_idx` argument, if an embedding's index is equal to `padding_idx` it is ignored, so it is not included in the reduction.
This PR does not add support for `padding_idx` for quantized or ONNX `EmbeddingBag` for opset10/11 (opset9 is supported). In these cases, an error is thrown if `padding_idx` is provided.
Fixes https://github.com/pytorch/pytorch/issues/3194
Pull Request resolved: https://github.com/pytorch/pytorch/pull/49237
Reviewed By: walterddr, VitalyFedyunin
Differential Revision: D26948258
Pulled By: jbschlosser
fbshipit-source-id: 3ca672f7e768941f3261ab405fc7597c97ce3dfc
