authored with codex
as discussed w/ @eellison , @ngimel ,~~~ just stashing this prototype here as performance doesn't look great on the hot path:~~~
```
Latest benchmark results:
Backend Shape (M,N,K) Cached Eager Delta
━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━ ━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━
cuBLAS (64,64,64) 14.807 µs 15.092 µs +0.284 µs / +1.92%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLAS (256,256,256) 14.904 µs 15.301 µs +0.397 µs / +2.66%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLAS (1024,1024,1024) 14.850 µs 15.363 µs +0.513 µs / +3.46%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLAS (32,2048,10944) 16.777 µs 17.626 µs +0.849 µs / +5.06%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLASLt (64,64,64) 14.781 µs 15.181 µs +0.399 µs / +2.70%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLASLt (256,256,256) 14.850 µs 15.099 µs +0.249 µs / +1.68%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLASLt (1024,1024,1024) 14.718 µs 14.792 µs +0.074 µs / +0.50%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLASLt (32,2048,10944) 16.715 µs 17.247 µs +0.532 µs / +3.18%
Paste-ready discussion summary—if this goes on GitHub, please add your own commentary around the quoted AI-generated material:
> AI-generated benchmark summary, provided for human review
>
> Benchmarked cached versus operation-scoped eager cuBLAS workspaces on an NVIDIA GB300, CC 10.3, CUDA 13.4. PyTorch was built for CC 10.0. Measurements are medians across five alternating cached/eager process trials using synchronized
> torch.utils.benchmark.Timer measurements of BF16 torch.mm.
>
> Eager workspace allocation added approximately 0.07–0.85 µs per operation. Relative overhead ranged from 0.5% to 5.1%, with the largest measured overhead on the legacy cuBLAS (M,N,K)=(32,2048,10944) regression shape.
>
> The cached implementation reproduced CUDA_ERROR_ILLEGAL_ADDRESS when separately captured graphs reused a workspace on the same stream. The eager implementation passed that reproducer.
>
> Given that most of the eager overhead appears to come from caching-allocator allocation/free bookkeeping, a simpler alternative under consideration is to retain the existing workspace cache but explicitly clear or isolate cached workspaces at
> graph-capture boundaries.
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/194311
Approved by: https://github.com/ngimel, https://github.com/eellison
authored with codex
as discussed w/ @eellison , @ngimel ,~~~ just stashing this prototype here as performance doesn't look great on the hot path:~~~
```
Latest benchmark results:
Backend Shape (M,N,K) Cached Eager Delta
━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━ ━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━
cuBLAS (64,64,64) 14.807 µs 15.092 µs +0.284 µs / +1.92%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLAS (256,256,256) 14.904 µs 15.301 µs +0.397 µs / +2.66%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLAS (1024,1024,1024) 14.850 µs 15.363 µs +0.513 µs / +3.46%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLAS (32,2048,10944) 16.777 µs 17.626 µs +0.849 µs / +5.06%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLASLt (64,64,64) 14.781 µs 15.181 µs +0.399 µs / +2.70%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLASLt (256,256,256) 14.850 µs 15.099 µs +0.249 µs / +1.68%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLASLt (1024,1024,1024) 14.718 µs 14.792 µs +0.074 µs / +0.50%
────────── ────────────────── ─────────── ─────────── ────────────────────
cuBLASLt (32,2048,10944) 16.715 µs 17.247 µs +0.532 µs / +3.18%
Paste-ready discussion summary—if this goes on GitHub, please add your own commentary around the quoted AI-generated material:
> AI-generated benchmark summary, provided for human review
>
> Benchmarked cached versus operation-scoped eager cuBLAS workspaces on an NVIDIA GB300, CC 10.3, CUDA 13.4. PyTorch was built for CC 10.0. Measurements are medians across five alternating cached/eager process trials using synchronized
> torch.utils.benchmark.Timer measurements of BF16 torch.mm.
>
> Eager workspace allocation added approximately 0.07–0.85 µs per operation. Relative overhead ranged from 0.5% to 5.1%, with the largest measured overhead on the legacy cuBLAS (M,N,K)=(32,2048,10944) regression shape.
>
> The cached implementation reproduced CUDA_ERROR_ILLEGAL_ADDRESS when separately captured graphs reused a workspace on the same stream. The eager implementation passed that reproducer.
>
> Given that most of the eager overhead appears to come from caching-allocator allocation/free bookkeeping, a simpler alternative under consideration is to retain the existing workspace cache but explicitly clear or isolate cached workspaces at
> graph-capture boundaries.
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/194311
Approved by: https://github.com/ngimel, https://github.com/eellison
The fix-cpp-build branch is a major restructuring that:
- Drops Exhale (which auto-generated API docs but caused OOM in CI) in favor of hand-written RST pages using Breathe directives
- Reorganizes notes into a hierarchical API reference with proper categories
- Adds new content: XPU docs, stable ABI docs, DeviceGuard/Stream docs
- Always runs Doxygen instead of gating on an env var
Pull Request resolved: https://github.com/pytorch/pytorch/pull/174096
Approved by: https://github.com/janeyx99, https://github.com/seemethere, https://github.com/albanD
Eddie Yan