mirror of
https://github.com/pytorch/pytorch.git
synced 2026-10-28 18:56:51 +08:00
Eddie Yan
a2bf3a697d
authored with codex as discussed w/ @eellison , @ngimel ,~~~ just stashing this prototype here as performance doesn't look great on the hot path:~~~ ``` Latest benchmark results: Backend Shape (M,N,K) Cached Eager Delta ━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━ ━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━ cuBLAS (64,64,64) 14.807 µs 15.092 µs +0.284 µs / +1.92% ────────── ────────────────── ─────────── ─────────── ──────────────────── cuBLAS (256,256,256) 14.904 µs 15.301 µs +0.397 µs / +2.66% ────────── ────────────────── ─────────── ─────────── ──────────────────── cuBLAS (1024,1024,1024) 14.850 µs 15.363 µs +0.513 µs / +3.46% ────────── ────────────────── ─────────── ─────────── ──────────────────── cuBLAS (32,2048,10944) 16.777 µs 17.626 µs +0.849 µs / +5.06% ────────── ────────────────── ─────────── ─────────── ──────────────────── cuBLASLt (64,64,64) 14.781 µs 15.181 µs +0.399 µs / +2.70% ────────── ────────────────── ─────────── ─────────── ──────────────────── cuBLASLt (256,256,256) 14.850 µs 15.099 µs +0.249 µs / +1.68% ────────── ────────────────── ─────────── ─────────── ──────────────────── cuBLASLt (1024,1024,1024) 14.718 µs 14.792 µs +0.074 µs / +0.50% ────────── ────────────────── ─────────── ─────────── ──────────────────── cuBLASLt (32,2048,10944) 16.715 µs 17.247 µs +0.532 µs / +3.18% Paste-ready discussion summary—if this goes on GitHub, please add your own commentary around the quoted AI-generated material: > AI-generated benchmark summary, provided for human review > > Benchmarked cached versus operation-scoped eager cuBLAS workspaces on an NVIDIA GB300, CC 10.3, CUDA 13.4. PyTorch was built for CC 10.0. Measurements are medians across five alternating cached/eager process trials using synchronized > torch.utils.benchmark.Timer measurements of BF16 torch.mm. > > Eager workspace allocation added approximately 0.07–0.85 µs per operation. Relative overhead ranged from 0.5% to 5.1%, with the largest measured overhead on the legacy cuBLAS (M,N,K)=(32,2048,10944) regression shape. > > The cached implementation reproduced CUDA_ERROR_ILLEGAL_ADDRESS when separately captured graphs reused a workspace on the same stream. The eager implementation passed that reproducer. > > Given that most of the eager overhead appears to come from caching-allocator allocation/free bookkeeping, a simpler alternative under consideration is to retain the existing workspace cache but explicitly clear or isolate cached workspaces at > graph-capture boundaries. ``` Pull Request resolved: https://github.com/pytorch/pytorch/pull/194311 Approved by: https://github.com/ngimel, https://github.com/eellison