mirror of
https://github.com/pytorch/pytorch.git
synced 2026-10-28 18:56:51 +08:00
Svetlana Karslioglu
19da3da3d3
The fix-cpp-build branch is a major restructuring that: - Drops Exhale (which auto-generated API docs but caused OOM in CI) in favor of hand-written RST pages using Breathe directives - Reorganizes notes into a hierarchical API reference with proper categories - Adds new content: XPU docs, stable ABI docs, DeviceGuard/Stream docs - Always runs Doxygen instead of gating on an env var Pull Request resolved: https://github.com/pytorch/pytorch/pull/174096 Approved by: https://github.com/janeyx99, https://github.com/seemethere, https://github.com/albanD
2.7 KiB
2.7 KiB
myst
| myst | ||||||
|---|---|---|---|---|---|---|
|
Normalization Layers
Normalization layers stabilize and accelerate training by normalizing intermediate activations. They help with gradient flow and allow higher learning rates.
- BatchNorm: Normalizes across batch dimension; most common in CNNs
- InstanceNorm: Normalizes each sample independently; popular in style transfer
- LayerNorm: Normalizes across feature dimension; standard in transformers
- GroupNorm: Normalizes within groups of channels; works with small batches
- LocalResponseNorm: Lateral inhibition inspired by neuroscience (less common today)
BatchNorm1d / BatchNorm2d / BatchNorm3d
:members:
:undoc-members:
:members:
:undoc-members:
:members:
:undoc-members:
:members:
:undoc-members:
:members:
:undoc-members:
:members:
:undoc-members:
Example:
auto bn = torch::nn::BatchNorm2d(
torch::nn::BatchNorm2dOptions(64) // num_features
.eps(1e-5)
.momentum(0.1)
.affine(true)
.track_running_stats(true));
InstanceNorm1d / InstanceNorm2d / InstanceNorm3d
:members:
:undoc-members:
:members:
:undoc-members:
:members:
:undoc-members:
:members:
:undoc-members:
:members:
:undoc-members:
:members:
:undoc-members:
LayerNorm
:members:
:undoc-members:
:members:
:undoc-members:
Example:
auto ln = torch::nn::LayerNorm(
torch::nn::LayerNormOptions({768})); // normalized_shape
GroupNorm
:members:
:undoc-members:
:members:
:undoc-members:
Example:
auto gn = torch::nn::GroupNorm(
torch::nn::GroupNormOptions(32, 256)); // num_groups, num_channels
LocalResponseNorm
:members:
:undoc-members:
:members:
:undoc-members: