Files
IvanKobzarev b64c98df16 [native_dsl] Add nvmath cublasLt grouped GEMM for _foreach_mm (#185530)
## Summary
Pure Python `_foreach_mm` override using nvmath cublasLt grouped GEMM bindings. Zero C++ changes, no data stacking — passes device pointer arrays directly to `cublasLtMatmul` with grouped matrix layouts.
- Caches cublasLt descriptors, layouts, heuristic-selected algorithm per `(M, N, K, G, strides, device)` shape key
- On repeated calls (Muon optimizer pattern), only pointer values are updated via pinned-memory async H2D
- Supports both row-major and column-major input strides
- Falls back gracefully when nvmath is unavailable
### Dispatch logic
```
nvmath available (nvmath-python installed, cublasLt >= 13.2 runtime)?
|
|-- YES --> nvmath cublasLt grouped GEMM (cached, no data copy)
|
|-- NO  --> min(M, N, K) >= 2048?
            |
            |-- YES --> [torch.mm(a, b) for ...] (each cuBLAS mm saturates GPU)
            |
            |-- NO  --> torch.stack -> _grouped_mm (CUTLASS 3D path)
```
### Files changed
- `torch/_native/ops/foreach_mm/impl.py` — dispatch override: cond validation, nvmath path with shape-keyed cache, stack fallback, loop fallback for large dims
- `torch/_native/ops/foreach_mm/nvmath_impl.py` — `ForeachMMCublasLt` class: cached cublasLt descriptors, pinned async H2D pointer transfer, grouped matrix layouts
- `torch/_native/ops/foreach_mm/__init__.py` — registers the override
- `torch/_native/ops/__init__.py` — adds `foreach_mm` to the native ops package
## Benchmarks
**Hardware:** NVIDIA H100 95 GB (isolated GPU, 0% utilization)
**Method:** CUDA events, 30 warmup, 300 iterations, median reported, bf16
### Standard shapes
| Shape | G | cutlass C++ (no desc cache) | cublaslt C++ (no desc cache) | nvmath (desc cached) | loop mm | nvmath / cutlass | nvmath / loop |
|:------|--:|--------:|--------:|--------:|--------:|-------:|-------:|
| 128^3 | 8 | 0.074 | 0.074 | **0.037** | 0.072 | **0.50x** | 0.51x |
| 256^3 | 8 | 0.075 | 0.075 | **0.038** | 0.073 | **0.51x** | 0.52x |
| 512^3 | 8 | 0.075 | 0.076 | **0.038** | 0.073 | **0.51x** | 0.52x |
| 512^3 | 64 | 0.209 | 0.209 | **0.094** | 0.537 | **0.45x** | **0.18x** |
| 512^3 | 128 | 0.379 | 0.374 | **0.163** | 1.120 | **0.43x** | **0.15x** |
| 1024^3 | 8 | 0.090 | 0.090 | **0.058** | 0.073 | **0.64x** | 0.79x |
| 1024^3 | 64 | 0.399 | 0.402 | **0.295** | 0.542 | **0.74x** | **0.54x** |
| 1024^3 | 128 | 0.785 | 0.789 | **0.571** | 1.058 | **0.73x** | **0.54x** |
| 2048^3 | 8 | 0.281 | 0.286 | **0.264** | 0.263 | **0.94x** | 1.00x |
| 2048^3 | 64 | 2.018 | 2.039 | **1.981** | 2.072 | **0.98x** | 0.96x |
| 2048^3 | 112 | 3.520 | 3.550 | **3.452** | 3.676 | **0.98x** | 0.94x |
### Muon optimizer shapes — dim=4096
| Shape | G | cutlass C++ | cublaslt C++ | nvmath | loop mm | nvmath / cutlass | Description |
|:------|--:|--------:|--------:|--------:|--------:|-------:|:------------|
| 4096^3 | 16 | 3.897 | 3.897 | **3.856** | 3.780 | **0.99x** | Attention weights (1 type x 16 layers) |
| 4096^3 | 64 | 15.659 | 15.576 | **15.350** | 15.158 | **0.98x** | Attention weights (all 4 types x 16 layers) |
| 4096^3 | 112 | 27.056 | 27.060 | **26.868** | 26.679 | **0.99x** | All non-embedding square weights |
| 4096 x 11008 | 16 | 10.153 | 10.150 | **10.097** | 9.929 | **0.99x** | MLP gate/up (SwiGLU) |
| 11008 x 4096 | 16 | 10.919 | 10.917 | 10.869 | 10.718 | 1.00x | MLP down |
| 4096 x 16384 | 16 | 14.997 | 15.027 | **14.941** | 14.649 | **1.00x** | MLP l1 (4x ffn) |
| 16384 x 4096 | 16 | 16.763 | 16.780 | 16.689 | 15.493 | 1.00x | MLP l2 (4x ffn) |
### Why nvmath is 2x faster than C++ at small shapes
The C++ `_foreach_mm` (both CUTLASS and cublasLt backends) recreates cublasLt descriptors, grouped layouts, and runs the heuristic algorithm search on every call. Per-call overhead breakdown:
| Component | C++ (per call) | nvmath cached (per call) |
|:----------|:--------------|:------------------------|
| TensorList validation | ~5 us | 0 (Python cond) |
| G separate output allocations | ~15 us | ~2 us (one alloc) |
| Pointer fill + async H2D | ~8 us | ~20 us (Python loop) |
| Descriptor create/destroy | ~10 us | 0 (cached) |
| **Heuristic search** | **~35 us** | **0 (cached)** |
| Kernel launch | ~8 us | ~8 us |
| **Total** | **~81 us** | **~30 us** |
The heuristic search (~35 us) is the single largest source of overhead in the C++ path and is completely eliminated by caching.
### When grouped GEMM beats naive loop
| Regime | Grouped GEMM vs loop |
|:-------|:--------------------|
| dim <= 1024, G >= 2 | Grouped wins (2-7x for G >= 64) |
| dim = 2048, G >= 16 | Grouped wins (~5% faster) |
| dim >= 4096 | Loop matches or wins (each mm saturates GPU) |
### Memory overhead
- Shared workspace: 32 MB (one per process, matches C++ cublasLt path)
- Per cached shape entry: ~3 KB GPU + ~1.5 KB pinned host
- Typical Muon usage (3-5 shapes): ~32 MB total
## Test plan
Correctness verified against `torch.mm` loop reference for:
- Square shapes: 64^2 to 4096^2
- Non-square MLP shapes: 4096x11008, 4096x16384
- All 4 stride combinations: row/row, row/col, col/row, col/col
- Repeated calls with same shape (cache correctness)
- fp32 dtype rejection (ValueError)
- Fallback when nvmath unavailable (ImportError / FunctionNotFoundError)
```bash
CUDA_VISIBLE_DEVICES=1 \
LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH \
python agent_space/test_foreach_mm_nvmath.py -v
```
Authored with Claude.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/185530
Approved by: https://github.com/slayton58
2026-06-05 19:18:52 +00:00

70 lines
2.0 KiB
Python

import importlib
import importlib.metadata
import os
from functools import cache
from torch._vendor.packaging import version as _packaging_version
@cache
def check_native_jit_disabled() -> bool:
"""
Single point to check if native DSL ops are disabled globally,
checked via:
TORCH_DISABLE_NATIVE_JIT=1
"""
return int(os.getenv("TORCH_DISABLE_NATIVE_JIT", 0)) == 1
def _unavailable_reason(deps: list[tuple[str, str]]) -> None | str:
"""
Check availability of required packages - cuteDSL & deps,
informing user what (if anything) is missing
NOTE: Doesn't actually import anything.
"""
for package_name, module_name in deps:
# find_spec raises ModuleNotFoundError for dotted names (e.g.
# "nvmath.bindings") when the parent package is absent.
try:
found = importlib.util.find_spec(module_name) is not None
except (ModuleNotFoundError, ValueError):
found = False
if not found:
return (
f"missing optional dependency `{package_name}` "
f"(importlib.util.find_spec({package_name}) failed)"
)
return None
def _available_version(package: str) -> _packaging_version.Version | None:
"""
Get the installed version of a package as (major, minor, patch).
Handles pre-release suffixes like "0.7.0rc1" or "3.1.0.post1" by
stripping non-numeric tails from each component. Returns None on
parse failure.
"""
try:
version = importlib.metadata.version(package)
except importlib.metadata.PackageNotFoundError:
return None
try:
v = _packaging_version.parse(version)
except _packaging_version.InvalidVersion:
return None
return v
@cache
def check_native_version_skip() -> bool:
"""
Single point to check if native DSL version gating should be skipped,
checked via:
TORCH_NATIVE_SKIP_VERSION_CHECK=1
"""
return int(os.getenv("TORCH_NATIVE_SKIP_VERSION_CHECK", 0)) == 1