mirror of
https://github.com/pytorch/pytorch.git
synced 2026-10-28 18:56:51 +08:00
## Summary
Pure Python `_foreach_mm` override using nvmath cublasLt grouped GEMM bindings. Zero C++ changes, no data stacking — passes device pointer arrays directly to `cublasLtMatmul` with grouped matrix layouts.
- Caches cublasLt descriptors, layouts, heuristic-selected algorithm per `(M, N, K, G, strides, device)` shape key
- On repeated calls (Muon optimizer pattern), only pointer values are updated via pinned-memory async H2D
- Supports both row-major and column-major input strides
- Falls back gracefully when nvmath is unavailable
### Dispatch logic
```
nvmath available (nvmath-python installed, cublasLt >= 13.2 runtime)?
|
|-- YES --> nvmath cublasLt grouped GEMM (cached, no data copy)
|
|-- NO --> min(M, N, K) >= 2048?
|
|-- YES --> [torch.mm(a, b) for ...] (each cuBLAS mm saturates GPU)
|
|-- NO --> torch.stack -> _grouped_mm (CUTLASS 3D path)
```
### Files changed
- `torch/_native/ops/foreach_mm/impl.py` — dispatch override: cond validation, nvmath path with shape-keyed cache, stack fallback, loop fallback for large dims
- `torch/_native/ops/foreach_mm/nvmath_impl.py` — `ForeachMMCublasLt` class: cached cublasLt descriptors, pinned async H2D pointer transfer, grouped matrix layouts
- `torch/_native/ops/foreach_mm/__init__.py` — registers the override
- `torch/_native/ops/__init__.py` — adds `foreach_mm` to the native ops package
## Benchmarks
**Hardware:** NVIDIA H100 95 GB (isolated GPU, 0% utilization)
**Method:** CUDA events, 30 warmup, 300 iterations, median reported, bf16
### Standard shapes
| Shape | G | cutlass C++ (no desc cache) | cublaslt C++ (no desc cache) | nvmath (desc cached) | loop mm | nvmath / cutlass | nvmath / loop |
|:------|--:|--------:|--------:|--------:|--------:|-------:|-------:|
| 128^3 | 8 | 0.074 | 0.074 | **0.037** | 0.072 | **0.50x** | 0.51x |
| 256^3 | 8 | 0.075 | 0.075 | **0.038** | 0.073 | **0.51x** | 0.52x |
| 512^3 | 8 | 0.075 | 0.076 | **0.038** | 0.073 | **0.51x** | 0.52x |
| 512^3 | 64 | 0.209 | 0.209 | **0.094** | 0.537 | **0.45x** | **0.18x** |
| 512^3 | 128 | 0.379 | 0.374 | **0.163** | 1.120 | **0.43x** | **0.15x** |
| 1024^3 | 8 | 0.090 | 0.090 | **0.058** | 0.073 | **0.64x** | 0.79x |
| 1024^3 | 64 | 0.399 | 0.402 | **0.295** | 0.542 | **0.74x** | **0.54x** |
| 1024^3 | 128 | 0.785 | 0.789 | **0.571** | 1.058 | **0.73x** | **0.54x** |
| 2048^3 | 8 | 0.281 | 0.286 | **0.264** | 0.263 | **0.94x** | 1.00x |
| 2048^3 | 64 | 2.018 | 2.039 | **1.981** | 2.072 | **0.98x** | 0.96x |
| 2048^3 | 112 | 3.520 | 3.550 | **3.452** | 3.676 | **0.98x** | 0.94x |
### Muon optimizer shapes — dim=4096
| Shape | G | cutlass C++ | cublaslt C++ | nvmath | loop mm | nvmath / cutlass | Description |
|:------|--:|--------:|--------:|--------:|--------:|-------:|:------------|
| 4096^3 | 16 | 3.897 | 3.897 | **3.856** | 3.780 | **0.99x** | Attention weights (1 type x 16 layers) |
| 4096^3 | 64 | 15.659 | 15.576 | **15.350** | 15.158 | **0.98x** | Attention weights (all 4 types x 16 layers) |
| 4096^3 | 112 | 27.056 | 27.060 | **26.868** | 26.679 | **0.99x** | All non-embedding square weights |
| 4096 x 11008 | 16 | 10.153 | 10.150 | **10.097** | 9.929 | **0.99x** | MLP gate/up (SwiGLU) |
| 11008 x 4096 | 16 | 10.919 | 10.917 | 10.869 | 10.718 | 1.00x | MLP down |
| 4096 x 16384 | 16 | 14.997 | 15.027 | **14.941** | 14.649 | **1.00x** | MLP l1 (4x ffn) |
| 16384 x 4096 | 16 | 16.763 | 16.780 | 16.689 | 15.493 | 1.00x | MLP l2 (4x ffn) |
### Why nvmath is 2x faster than C++ at small shapes
The C++ `_foreach_mm` (both CUTLASS and cublasLt backends) recreates cublasLt descriptors, grouped layouts, and runs the heuristic algorithm search on every call. Per-call overhead breakdown:
| Component | C++ (per call) | nvmath cached (per call) |
|:----------|:--------------|:------------------------|
| TensorList validation | ~5 us | 0 (Python cond) |
| G separate output allocations | ~15 us | ~2 us (one alloc) |
| Pointer fill + async H2D | ~8 us | ~20 us (Python loop) |
| Descriptor create/destroy | ~10 us | 0 (cached) |
| **Heuristic search** | **~35 us** | **0 (cached)** |
| Kernel launch | ~8 us | ~8 us |
| **Total** | **~81 us** | **~30 us** |
The heuristic search (~35 us) is the single largest source of overhead in the C++ path and is completely eliminated by caching.
### When grouped GEMM beats naive loop
| Regime | Grouped GEMM vs loop |
|:-------|:--------------------|
| dim <= 1024, G >= 2 | Grouped wins (2-7x for G >= 64) |
| dim = 2048, G >= 16 | Grouped wins (~5% faster) |
| dim >= 4096 | Loop matches or wins (each mm saturates GPU) |
### Memory overhead
- Shared workspace: 32 MB (one per process, matches C++ cublasLt path)
- Per cached shape entry: ~3 KB GPU + ~1.5 KB pinned host
- Typical Muon usage (3-5 shapes): ~32 MB total
## Test plan
Correctness verified against `torch.mm` loop reference for:
- Square shapes: 64^2 to 4096^2
- Non-square MLP shapes: 4096x11008, 4096x16384
- All 4 stride combinations: row/row, row/col, col/row, col/col
- Repeated calls with same shape (cache correctness)
- fp32 dtype rejection (ValueError)
- Fallback when nvmath unavailable (ImportError / FunctionNotFoundError)
```bash
CUDA_VISIBLE_DEVICES=1 \
LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH \
python agent_space/test_foreach_mm_nvmath.py -v
```
Authored with Claude.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/185530
Approved by: https://github.com/slayton58
70 lines
2.0 KiB
Python
70 lines
2.0 KiB
Python
import importlib
|
|
import importlib.metadata
|
|
import os
|
|
from functools import cache
|
|
|
|
from torch._vendor.packaging import version as _packaging_version
|
|
|
|
|
|
@cache
|
|
def check_native_jit_disabled() -> bool:
|
|
"""
|
|
Single point to check if native DSL ops are disabled globally,
|
|
checked via:
|
|
TORCH_DISABLE_NATIVE_JIT=1
|
|
"""
|
|
return int(os.getenv("TORCH_DISABLE_NATIVE_JIT", 0)) == 1
|
|
|
|
|
|
def _unavailable_reason(deps: list[tuple[str, str]]) -> None | str:
|
|
"""
|
|
Check availability of required packages - cuteDSL & deps,
|
|
informing user what (if anything) is missing
|
|
|
|
NOTE: Doesn't actually import anything.
|
|
"""
|
|
for package_name, module_name in deps:
|
|
# find_spec raises ModuleNotFoundError for dotted names (e.g.
|
|
# "nvmath.bindings") when the parent package is absent.
|
|
try:
|
|
found = importlib.util.find_spec(module_name) is not None
|
|
except (ModuleNotFoundError, ValueError):
|
|
found = False
|
|
if not found:
|
|
return (
|
|
f"missing optional dependency `{package_name}` "
|
|
f"(importlib.util.find_spec({package_name}) failed)"
|
|
)
|
|
return None
|
|
|
|
|
|
def _available_version(package: str) -> _packaging_version.Version | None:
|
|
"""
|
|
Get the installed version of a package as (major, minor, patch).
|
|
|
|
Handles pre-release suffixes like "0.7.0rc1" or "3.1.0.post1" by
|
|
stripping non-numeric tails from each component. Returns None on
|
|
parse failure.
|
|
"""
|
|
try:
|
|
version = importlib.metadata.version(package)
|
|
except importlib.metadata.PackageNotFoundError:
|
|
return None
|
|
|
|
try:
|
|
v = _packaging_version.parse(version)
|
|
except _packaging_version.InvalidVersion:
|
|
return None
|
|
|
|
return v
|
|
|
|
|
|
@cache
|
|
def check_native_version_skip() -> bool:
|
|
"""
|
|
Single point to check if native DSL version gating should be skipped,
|
|
checked via:
|
|
TORCH_NATIVE_SKIP_VERSION_CHECK=1
|
|
"""
|
|
return int(os.getenv("TORCH_NATIVE_SKIP_VERSION_CHECK", 0)) == 1
|