* the branch is now pointing to main and not to unify-resolve-plan
* Refactor expert parallelism registration and improve documentation
- Consolidated the registration of parallel styles into the `ParallelInterface` class for better organization.
- Updated the `expert_parallel.py` and `tensor_parallel.py` files to streamline the registration process of expert parallel strategies.
- Enhanced documentation for clarity regarding the usage of tensor parallel strategies and their registration.
- Removed redundant code related to `MoeExpertsParallel` and `MoeIdentityParallel` classes, which are now properly registered in the new structure.
* comment
* some models missed the tests EP mixin
* fix copyright
* Improve clarity and grammar in Auto Classes documentation
Improved wording and grammar in the Auto Classes documentation
to enhance readability without changing functionality.
* reviewer fixes
Clarified instructions for extending Auto classes with custom models and configurations.
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
openai/whisper-large-v3-turbo loads in float16 by default, while the
feature extractor emits float32, so the example raised
"Input type (float) and bias type (c10::Half) should be the same".
Fixes#47805
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Initial set of changes for Python>=3.11
* Few more ruff UP rule fixes
* from typing import Unpack
* Remove tomli
* Update doc references to py3.10
* Address few comments
* Address more sergereview comments
---------
Co-authored-by: Tarek Ziade <tarek@ziade.org>
* Add an FP8 backend to `QuantizedCache`
* Reorder the FP8 states for beam search
* Use an early return in `Fp8QuantizedLayer.reorder_cache`
* Measure the quantized cache backends accurately
The memory of the quantized states was under-reported: only `fp8` stores
them as a plain tensor, `hqq` uses a `(tensor, metadata)` tuple and quanto
a tensor subclass whose payload sits in its attributes, neither of which
`numel() * element_size()` accounts for.
Also report the KL divergence of the next-token distribution against the
bf16 one, as the matching token count saturates on short generations, and
copy the `cache_config` per call, which `generate` consumes.
* Reset the FP8 scales, and cover the FP8 backend in the quantized layer tests
* Address review on the FP8 quantized cache layer
- `fp8_max` becomes a class attribute next to `fp8_dtype`, instead of being
recomputed and stored on every instance. This matches how the FP8 constants
are already defined in `integrations/finegrained_fp8.py` and
`integrations/compressed_tensors.py`.
- `Fp8QuantizedLayer` now overrides `lazy_initialization` to calibrate the
scales, rather than doing it inline in the first `update` call.
- `_quantized_keys` and `_quantized_values` are initialized to `None` in
`QuantizedLayer.__init__`, so that they are always defined before the first
`update`. This was missing for the quanto and hqq layers as well.
- `.contiguous()` is now only called in `lazy_initialization`. The states
passed to `update` come straight from a transpose in the attention layer, so
they are genuinely non-contiguous during prefill, and `_quantize` propagates
that layout. Every subsequent append goes through a `cat`, whose output is
contiguous by construction, so the call is redundant there.
- `_cat_fp8` is removed in favour of a plain `torch.cat`, which handles
`float8_e4m3fn` natively on CPU and CUDA, for contiguous and non-contiguous
inputs alike (checked on torch 2.8 through 2.14).
* Trim the comments down to one line
* Store the FP8 scale alongside the states it quantizes
`_quantize` already returned a `(states, scale)` pair, but the two halves were
then split across `_quantized_keys` and a separate `_key_scale` attribute, and
zipped back together at every `_dequantize` call site.
Keeping the pair as the layer's single quantized payload matches what
`HQQQuantizedLayer` does with `(qtensor, meta)`, lets `_dequantize` be called on
`_quantized_keys` directly like the base class does, and removes the `reset`
override, since `QuantizedLayer.reset` already drops `_quantized_keys`.
It also keeps the scale's rank internal to `_quantize` / `_dequantize`, so a
future per-block backend can make it a tensor without changing the layer's
attributes.
* Say what the `.contiguous()` in `lazy_initialization` is actually for
Nothing in `_quantize` needs a contiguous input: on strided states it computes
the same scale and gives bit-identical dequantized values.
What the call is for is the layout of the stored buffer. The states handed to
`lazy_initialization` come from a transpose in the attention layer, while the
ones `update` appends come out of `cat`, and the two disagree: without the call
the buffer is laid out as `[B, S, H, D]` after the prefill and as `[B, H, S, D]`
from the first decode step on. Normalizing it once keeps the strides the same
for the buffer's whole life, which matters for the follow-up feeding these
states to FP8 attention kernels.
The test was building its states with `torch.randn(1, 2, 5, 8)`, so they were
already contiguous and the assertion held whether or not the call was there. It
now builds them the way the attention layer does.
* Address @vasqu's review
Drop the benchmark script: it belongs in the PR description rather than in the
code base.
Drop the `.contiguous()` in `lazy_initialization`. Nothing in `_quantize` needs
it, and nothing reads the quantized states while they are still strided, so the
layout only matters once something actually consumes the FP8 buffer. Better
added then, with a consumer to justify it.
Trim the class docstring down to what is specific to this backend, and the
`_quantize` and `reorder_cache` ones down to one point each.
The test now builds its states the way the attention layer does, so the strided
path is covered.
* One-line the docstrings and comments that fit on one line
* Keep the FP8 state initialization out of the shared quantized layer
* Split the quantized layer into a base and a group-wise class
The group-wise options only belong to the integer backends, so they move to
`GroupWiseQuantizedLayer` and `Fp8QuantizedLayer` no longer inherits them.
`QuantizedCache` now rejects them for the `fp8` backend instead of silently
replacing them.
* Note which subtree a future fp8 variation belongs to
Grouping along the head dimension keeps the states appendable, so it has no
use for the residual window `GroupWiseQuantizedLayer` exists for.
* Keep the group-wise signature on QuantizedLayer and name the new base
The split moved the group-wise options out of QuantizedLayer, which breaks
subclasses that pass them to super().__init__. Keep that signature where it
was and introduce BaseQuantizedLayer for what the backends share instead.
The vibevoice guard now skips on the new base, so FP8 layers keep being
skipped there.
* Declare the quantized states in the base layer, and group the quantized tests
The states were only assigned in `reset` and `lazy_initialization`, so a freshly
built layer had no such attribute at all.
The FP8 tests move next to the other quantized ones, and the beam reordering and
reset tests are parameterized over the backends instead of duplicated.
* Add UltraVBert: UltraBERT text encoder with a SigLIP vision encoder
* Cap UltraVBert page tiles and add its model doc
* Trim UltraVBert comments
* Document UltraVBert's image_hidden_states and image-processor kwargs, date its model doc
* Add UltraVBert integration tests on the released checkpoint
* Skip SDPA flash dispatch for UltraVBert's sliding-window text tower
* Describe UltraVBert as extending ModernVBert in its model doc
* Update docs/source/en/model_doc/ultravbert.md
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Load the receipt fixture with load_image in the UltraVBert usage example
* Split the UltraVBert usage example into AutoModel and Pipeline tabs
* Always RMS-normalize UltraVBert's image tokens and drop connector_norm
* Batch UltraVBert's capped resize, write its image processing out in full and default the tile cap to 27
* Reuse ModernVBert's MLM head layout and forwards in UltraVBert and test it with the VLM tester
* Address review: drop the PIL image processor, adopt ModernVBert task tests
* Drop SigLIP's unused pooling head from UltraVBert
* Default UltraVBert's vision config to no pooling head through init_kwargs
* last stuff
* fix model parallel bug for molmo2 model (#49816)
Signed-off-by: kaixuanliu <kaixuan.liu@intel.com>
* fix(audio): Route the ALM conversion scripts through from_pretrained (#48525)
fix: Route the ALM conversion scripts through from_pretrained
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
---------
Signed-off-by: kaixuanliu <kaixuan.liu@intel.com>
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
Co-authored-by: vasqu <antonprogamer@gmail.com>
Co-authored-by: kaixuanliu <kaixuan.liu@intel.com>
Co-authored-by: Harshal Janjani <harshaljanjani@gmail.com>
* flint model support
* address comments
* Use public Flint checkpoint in release integration
* Align Flint expected completion with validated checkpoint
* should have backend support
---------
Co-authored-by: Lovish Madaan <lm@recursive.com>
Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com>
* [pocket_tts] generate and tests: hooks for derived models
- PocketTTSGenerationMixin._placeholder_token_ids: the ids excluded from the text count
of the default frame budget, overridable by a model without a voice BOS.
- The voice-as-cache check looks for the voice placeholder instead of the voice BOS
(a voice always comes as both in Pocket TTS).
- The tester's training and voice-cache helpers, and the test class, use the tester's
model_class and audio_prompt_dim instead of PocketTTSForConditionalGeneration and
latent_dim.
* Add Gradium Phonon TTS (modular over Pocket TTS)
GradiumPhononTTS: Pocket TTS trained by Gradium. Differences, all in the modular file:
the voice prompt is made of 512-dim speaker-codec latents (`audio_prompt_dim`), a
learnt speaker vector is added to every audio latent (`embed_audio_latents`), no voice
BOS (config and processor). The generation mixin subclasses Pocket TTS's
(nemotron3_5_asr precedent, TRF009 allowlist). Conversion, the normalising chat
template and the processor tests follow.
Claude-Session: https://claude.ai/code/session_01ECprm731kAdBUko4MjrXDg
* [gradium_phonon_tts] normalising chat template, processor tests
The original runtime's `normalize_text` (5 languages, `lang` template variable, "none"
skips it) and `prepare_text_prompt`, ported to the chat template: matches the Rust
runtime's fixture (gradium-ai/xn-ptts@preview-0.4.0, 13 normalisation and 5 preparation
rows) and the Python port on 72k random strings, every language.
Claude-Session: https://claude.ai/code/session_01ECprm731kAdBUko4MjrXDg
* [gradium_phonon_tts] converter, Gradium's timestep norm, integration tests
- Converter: the model package (GGUF, tokenizer, voices) to the TTS checkpoint and a
decoder-only audio tokenizer (sliding window 251, exact GELU); the chat template;
the runtime's generation config. Sanity: 51 frames, first end-of-speech logit
-8.1250, as the Rust runtime.
- GradiumPhononTTSTimestepNorm: Gradium's runtime normalises the time embeddings with
the biased variance rescaled by sqrt((n - 1) / n) (Pocket TTS: the unbiased one), which
only differs in where eps sits but moved the free-running latents by 1e-3.
- No fade-in by default (Gradium's runtime applies none).
- Integration tests (Rust runtime, f32): single, batch, sampled, tokenization.
Claude-Session: https://claude.ai/code/session_01ECprm731kAdBUko4MjrXDg
* [gradium_phonon_tts] follow Pocket TTS: first audio position, voice items, training
- `<|audio_bos|>` closes the prompt (`audio_bos_token_id`, vocab 4003), as Pocket TTS.
- Chat template: the voice placeholder only with an `audio` item (empty for precomputed
latents), the training branch (assistant target audio); the normalisation unchanged.
- Processor: no voice BOS, voices and targets told apart by `audio_bos_token` alone.
- Config docstrings carry Pocket TTS's training fields; converter: the new token, the
training-only weights ignored, generation config returns the dict.
- Converter sanity unchanged (51 frames, first end-of-speech logit -8.1250, as Gradium's
runtime); integration 4/4 on CPU and GPU.
* [gradium_phonon_tts] follow Pocket TTS: flat audio tokenizer config, decoder_seanet
* [gradium_phonon_tts] conversion: latent_proj in the audio tokenizer
* [gradium_phonon_tts] follow Pocket TTS: loss_function, loss weighting on the model
* tmp
* tmp
* use pocket tts proc
* use pocket tts proc
* wip: uncommitted changes
* [gradium_phonon_tts] GradiumPhononTTSTokenizer (modular over PocketTTSTokenizer): text preparation out of the chat template
* [gradium_phonon_tts] tokenizer: normalizers for the character pass, data-driven number rewrite
* [gradium_phonon_tts] drop GradiumPhononTTSTokenizer: the runtime's text preparation as the normalizer of a versioned tokenizer file
* [gradium_phonon_tts] PocketTTSTokenizer does the capitalization: drop the per-letter normalizer steps
* [gradium_phonon_tts] back to TokenizersBackend: the capitalization as per-letter normalizer steps
* [gradium_phonon_tts] conversion: the runtime's tokenizer.json as it is, the placeholder names in tokenizer.5.0.0.json
* [gradium_phonon_tts] follow Pocket TTS: latent statistics in fp32
* [gradium_phonon_tts] follow Pocket TTS: return_tensors default in tokenizer_config.json
* [gradium_phonon_tts] num_speakers as a long tensor
* [gradium_phonon_tts] config: plain defaults for num_speakers and duration_delta_max_period
The modular converter cannot parse an `interval(...)(default=...)` override of an inherited config class.
* [gradium_phonon_tts] tests: no fixtures dir
Integration metadata from the bucket (as Pocket TTS), the chat template from the converted checkpoint. The Rust
runtime goldens (text normalisation, prepare_text_prompt) are dropped for now, to re-add from the bucket.
* [gradium_phonon_tts] no processor tests: it reuses PocketTTSProcessor, tested with Pocket TTS
* [gradium_phonon_tts] fix CI: ruff format, allow config attributes read by generate and the LSD loss
* [pocket_tts] cast the inputs of audio_prompt_proj / embed_audio_latents to the projection's dtype
Voices and generated latents are fp32 whatever the model's dtype, so a bf16 load
(e.g. dtype="auto" on a bf16 config) crashed in the projections.
* [gradium_phonon_tts] config dtype from the stored flow_lm weights (bf16)
- conversion: dtype="auto" now loads the TTS weights in the dtype they are stored in
- embed_audio_latents override: cast to the projection's dtype, as Pocket TTS
- integration tests: pin dtype=torch.float32 (Rust runtime f32 references)
* [processor] forward only the Hub access and placement kwargs to the audio tokenizer
The audio tokenizer lives in another repository, but `from_pretrained` forwarded every kwarg to it:
`revision="refs/pr/1"` looked for that ref in the audio tokenizer's repo, and `tokenizer=...` or another
sub-processor override reached the audio tokenizer model's `__init__`. It now gets `cache_dir`, `force_download`,
`proxies`, `token`, `local_files_only`, `device_map` and `dtype` only.
- Pocket TTS doc: no `encoder_config=None` on the processor any more (no longer forwarded)
- Pocket TTS processor test: the common `test_save_load_pretrained_additional_features` passes as is, a test that
`tokenizer=` and `revision` are not forwarded
* [gradium_phonon_tts] tokenizer normalizer: only the runtime's language-independent text preparation
The normalizer followed the runtime's English normalization as of 1 October, whatever the checkpoint's language. It now
does only what does not depend on the language, as ptts v0.4.0 does it: fold the characters the vocabulary would read as
raw bytes (typographic quotes and dashes, bullets, arrows, emoji, control characters, letters it lacks, subscripts), the
pause punctuation, then `prepare_text_prompt`. Numbers, amounts, symbols read aloud, addresses and the like are left to
the runtime's `ptts.normalize`, whose output these steps leave unchanged (checked on ~46k texts in five languages); on
text without such content they match the runtime exactly.
* [gradium_phonon_tts] doc: the released checkpoints, voices, batches and long text, text normalization
- Overview: the seven gradium/phonon-* checkpoints, voices as precomputed latents (each checkpoint's default voice and
the gradium/phonon-voices catalog)
- Usage tips, as Pocket TTS's: chat template and voice latents, decoding, the runtime's sampling defaults, bf16 (float32
on CPU to match the runtime), split_text
- Usage: a catalog voice; a batch of a long text split into chunks and another voice
- Text normalization: what the tokenizer does and does not, ptts for the rest
* [gradium_phonon_tts] conversion: no training fields in the saved config (training is not supported)
* [gradium_phonon_tts] integration tests: gradium/phonon-31m-en, goldens from Gradium's runtime (ptts 0.4.0)
The integration tests ran the pre-release checkpoint (eustlb/gradium-phonon-tts) against goldens from a pre-release
runtime. They now run gradium/phonon-31m-en (the commit that adds the transformers files) with voices of
gradium/phonon-voices, against goldens computed by ptts 0.4.0 from PyPI on the checkpoint and voice revisions it pins
(f32, temperature 0, one thread): https://gist.github.com/eustlb/b54c2ef3b6d0e92b20b23a974fbefad6
- single and batch: frame counts, then waveforms (AUDIO_ATOL 2e-3: up to 5.3e-4 on the CI A10G, 7.9e-4 on CPU)
- no sampled case and no per-step latents: the released runtime neither replays given noise nor returns latents
* add PocketTTS
* [processor] audio_tokenizer_kwargs, forwarded to the audio tokenizer's from_pretrained
* [processor] load the audio tokenizer in get_processor_dict, as before
* [pocket_tts] conversion: latent_proj in the audio tokenizer, frame-size padding in its feature extractor
* clean
* nit
* test cleaning
* loss update
* proc update
* [processor] drop audio_tokenizer_kwargs: only register PocketTTSAudioTokenizerModel as an audio tokenizer
* [pocket_tts] docs: place the processor's audio tokenizer with device_map
* [pocket_tts] place a generation step's latent position-wise, leaving torch_compilable_check untouched
* [pocket_tts] processor tests: chat template from the converted checkpoint, no .jinja fixtures
* [pocket_tts] drop the feature extractor tests, covered by the voice-cloning integration tests
* [pocket_tts] drop the torch_compilable_check test, its import_utils change is reverted
* test updates
* add comment
* nits
* [pocket_tts] docs: load the audio tokenizer without its encoder for cached voices
* [pocket_tts] drop the module docstrings that restate the file name
* add PocketTTSTokenizer
* [pocket_tts] tokenizer: text preparation as tokenizers normalizers, only the used steps
* [pocket_tts] drop PocketTTSTokenizer: the normalizer ships in tokenizer.json, the chat template capitalizes
* [pocket_tts] processor: split_text skips the markers the vocabulary lacks
* [pocket_tts] PocketTTSTokenizer: tokenizer.json's normalizer, then the capitalization
* [pocket_tts] back to TokenizersBackend: the capitalization as per-letter normalizer steps
* [pocket_tts] training-only state comment: its share of the weights
* [pocket_tts] latent statistics in fp32: in bf16, their EMA updates round away
- `_keep_in_fp32_modules_strict`: `latent_mean`, `latent_std`, and `freqs` (already built and saved in fp32)
- casts back to the input dtype after normalising / denormalising
- the EMA gate skips the `.item()` sync once the EMA is over
- common `test_keep_in_fp32_modules`: skip integer buffers, which are never cast
* [pocket_tts] processor: return_tensors default in tokenizer_config.json, no _defaults (TRF019)
* [pocket_tts] tests: readable voice cache loading
* [pocket_tts] feature extractor tests: frame-size padding, end_on_pause, get_num_frames
* [pocket_tts] drop unused Llama imports
* [pocket_tts] allow the LSD loss config attributes in check_config_attributes (read in loss/loss_lsd.py)
* add PocketTTSAudioTokenizer
* nit
* [pocket_tts_audio_tokenizer] flat config, modular over MimiConfig
* [pocket_tts_audio_tokenizer] follow Mimi: drop the sub_configs TODO
* [pocket_tts_audio_tokenizer] inherit Mimi's resnet block and SEANet encoder
* [mimi] latent_stride / latent_dim config properties for the downsample / upsample
* [modular] a modular attribute overrides an inherited method of the same name
* [pocket_tts_audio_tokenizer] inherit MimiEncoderModel / MimiDecoderModel
* [pocket_tts_audio_tokenizer] regenerate after Mimi's output capturing
* [pocket_tts_audio_tokenizer] inherit Mimi's encoder / decoder forward
* [mimi] generic docstrings for the encoder / decoder halves, quantizer no-split on MimiModel
* [pocket_tts_audio_tokenizer] half configs, submodules, outputs and config defaults from Mimi
* nit
* revert modular converted change
* [pocket_tts_audio_tokenizer] latent_stride / latent_dim fields replace Mimi's properties
* clean tests
* update tests
* nit
* [pocket_tts_audio_tokenizer] regenerate after Mimi's device-safe length helpers
* [pocket_tts_audio_tokenizer] regenerate after Mimi's sub_configs comment
* [pocket_tts_audio_tokenizer] work around the modular converter on chained field calls
* [pocket_tts_audio_tokenizer] drop the noqa'd property stubs: modular class attributes now shadow inherited methods
A config field in modular (latent_stride, latent_dim) replaces the parent's property of the same name, as in Python, so the
raise AttributeError stubs and their noqa: F811 (TRF035) are no longer needed. No other modular file changes.
* [pocket_tts_audio_tokenizer] model card: update date, link Pocket TTS to the Hub (its doc page lands in #49589)
* draft
* [mimi] encoder/decoder: nullable halves in MimiConfig, quantizer moved to MimiModel
* [mimi] drop core edits for the quantizer tie, no longer needed
* [mimi] drop the sub_configs TODO
* [mimi] slimmer halves tests
* [mimi] neutral _conv1d_layer_names, so derived codecs inherit the SEANet encoder
* [mimi] record outputs of MimiEncoderModel / MimiDecoderModel with capture_outputs
* [mimi] device-safe length helpers under device_map
* [mimi] comment why the encoder / decoder configs are not sub_configs
* Draft PR to fix thinking -> reasoning_content
* Cleanup
* More function simplification
* Add tests for reasoning field aliasing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* init: Add all files
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Compose the thinker base models instead of inheriting them
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* fix: Restore thinker encoder delegation, PEFT alias and CG docstrings
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Align the thinker base models with Qwen2-VL
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* refactor: Drop the no-mask get_rope_index branches and reword the skip reason
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
* test: Override test_inputs_embeds_matches_input_ids for the Omni thinkers
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
---------
Signed-off-by: Harshal Janjani <harshaljanjani@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Raushan Turganbay <raushan@huggingface.co>
* TP: keep colwise/rowwise for linear layers, give embeddings their own styles
`ColwiseParallel` and `RowwiseParallel` no longer special-case `nn.Embedding`; they now require a linear layer.
Embeddings get `EmbeddingRowwiseParallel` (`embedding_rowwise`, vocab-parallel) and `EmbeddingColwiseParallel`
(`embedding_colwise`, hidden-dim parallel, used by qwen4_exp whose ngram table loads as vocab chunks).
Doge's MoE embeddings move to `embedding_rowwise`, and gemma4's vision plan targets the inner `.linear`
of `Gemma4ClippableLinear`, which holds the weights.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* TP: split a linear layer unevenly when its width does not divide by the TP size
`DTensor.from_local` assumes even shards, so colwise outputs and rowwise inputs are wrapped with the full
shape taken from the module's `out_features` / `in_features`. The colwise divisibility check goes away, and
the TP tester keeps the tiny vocab of 99 so `lm_head` exercises the uneven split end to end.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* TP: one EmbeddingParallel style for both embedding shardings
`embedding_rowwise` and `embedding_colwise` become `EmbeddingParallel(shard_dim=0)` and
`EmbeddingParallel(shard_dim=1)` instead of two classes, and a hidden-sharded embedding no longer
needs a hidden size divisible by the TP size: its local output is wrapped with
`_from_local_last_dim_shard`, like the linear layers'. The helper always shards the last dim, so it
drops its placement argument.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* TP: reject packed weights whose chunks don't divide by the number of ranks
The strided split of a packed weight (an MoE experts' fused gate_up_proj) can't hold uneven
slices, so fail up front with a clear error instead of building a misshapen DTensor.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* TP: inline the packed divisibility check into both packed styles
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* TP: let packed weights shard unevenly too
The packed-weight divisibility check was unneeded: DTensor's strided split (with
`src_data_rank=None`, as transformers distributes) and the checkpoint loader both give each rank
matching ceil-sized slices of every packed chunk, so an uneven MoE `gate_up_proj` works. Verified
with a TP forward and backward of a tiny Qwen3-MoE whose `moe_intermediate_size=36` splits 8/8/8/8/4
over 5 ranks. The TP mixin's packed-gradient slicing now uses the same ceil-sized shards.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* TP: shard every model's input embeddings with `embedding_rowwise`
Every TP plan now vocab-shards the model's input embeddings: 89 plans with a plain `nn.Embedding`,
the scaled-embedding family (Gemma*, VaultGemma, MiniCPM3, NeoMME, T5Gemma*), Qwen4-Exp's
`embed_tokens`, and the models whose only plan was a colwise `lm_head` (AFMoE, DBRX, GPT-OSS,
HunYuan, LFM2, PhiMoE). Replicated, the embedding cost `vocab x hidden` on every rank, and a tied
`lm_head` sharded on the vocabulary did not match the embedding weight it shares.
`test_tp_plan_shards_input_embeddings` keeps every model with a TP plan sharding its embeddings.
The models that now get TP tests for the first time skip what can't pass: PhiMoE's backward (it
routes by Gumbel sampling in training) and DBRX's tied forward (it doesn't support tying).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* TP: shard `embed_tokens` by default instead of in every plan
Drop the `tie_word_embeddings` gate on the config-level `embedding_rowwise` default (explicit plan entries win), remove the per-config `embed_tokens` entries, and leave the lm_head-only models out of the embedding rollout; `test_tp_plan_shards_input_embeddings` now only checks decoder plans.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* TP: add the input-embedding default in `init_parallel_plans`
The config-level default added `embed_tokens` to every plan, including vision towers and models whose embedding has another name, which `test_tp_plan_matches_params` rejects. The model now adds `embed_tokens: embedding_rowwise` only when its base model has an `nn.Embedding` of that name; composite models pick it up from their text sub-model's plan. The tied-embeddings config default is left as on main for backward compatibility.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* TP: drop the tied-embeddings config default
`init_parallel_plans` now adds the same `embed_tokens` entry, tied or not, so the config-level one only duplicated it (and overrode a plan's own `embed_tokens` style).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Apply suggestion from @IlyasMoutawwakil
* Apply suggestion from @IlyasMoutawwakil
* fix nemotron H
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs: add Google TPU hardware guide
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* review(doc): change torch_tpu ->
* review(doc): drop attn_implementation in tpu
* review(doc): fix last sentences for tpu
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add MOSS-Transcribe-Diarize model.
* mlinter
* mlinter
* processing
* processing
* vibe voice parent
* vibe voice parent
* update other asr models with utils
* clean up
* default chat template in new checkpoint
* glmasr model parent
* feedback
* apply feedback - vecotirze chunking, use padding_mask
* add conversion script
* Restore audio_utils language/chat-content helpers dropped by rebase
The "feedback" commit removed make_audio_chat_template_content,
resolve_language, and prepare_language_inputs from audio_utils.py,
but other ASR models (qwen3_asr, canary, fun_asr_nano, audioflamingo3)
still depend on them. qwen3_asr had gained local resolve_language /
_prepare_language_inputs duplicates that were never actually wired
up, so its real call sites broke with a NameError. Restore the
shared helpers and drop the dead local duplicates.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Revert "Restore audio_utils language/chat-content helpers dropped by rebase"
This reverts commit 6e508e1ccb.
* fix rebase
* feedback - merge audio utils, keyword support, docs
* repo clean
* feedback
* fix loc of util
* drop glm as a parent
* move utils around
* minor fixes
* mlinter and processing test
* doc changes lost
* fixtures gists
* drop defaults
* Remove some comments from code review
Co-authored-by: Eric Bezzam <4757445+ebezzam@users.noreply.github.com>
* apply feedback - clean ups / nits
* moss feature extractor
* whisper encoder pass through
* revert audio_utils edit, will do in seperate PR
* test_feature_extraction
* md torch compile
* ruff
* ruff
* use qwen2audioencoder instead of whisper for now, match voxtral
* update conversion script
* modular file feedback + related models from qwen2 fixes, update fixtures
* update tests
* check repo
* small config fix
* update subconfigs
* repoint to hf-audio repo
* update tp_plan
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Eric Bezzam <4757445+ebezzam@users.noreply.github.com>
* Integrate the UltraBERT architecture
* Simplify UltraBERT attention mask handling
* Add UltraBERT token classification and question answering heads
* Simplify AutoModel snippets, add pipeline in docs
* Add embeddings pipeline too
* Refactor to use layer types and new standard RoPE configuration
* Rely on more modular superclasses and use "model" base prefix
* Simplify MLM w. ModernBERT superclass, simplify pred. head
* Move simple CrossEncoder test model to cross-encoder-testing
* Simplify UltraBERT tests and use MemoryCleanupMixin
* Replace Gemma3Rotary with ModernBertRotary
* Assume BC configs are correct
* added +1 comment
* Expand Attention to kill caching, add comments + spacing
* Shrink and simplify tests
* Avoid WeightRenaming on full "ultrabert", just on specific classes now
* Skip test_sdpa_can_dispatch_on_flash as sliding mask is mandatory
* fix(distributed): keep enable_expert_parallel as EP alone until its removal
Since v5.19, `enable_expert_parallel=True` became an alias for
`ep_size=tp_size`, which also applies the TP plan: unchanged v5.18 code
now shards attention and crashes on models with fewer KV heads than
`tp_size`. Without `ep_size`, the flag applies the EP plan alone again,
as before v5.19, and `model.tp_plan` returns it.
Fixes#49628
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* review: remove enable_expert_parallel in v5.21, after the release that fixes it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* review: drop the version history from the docstrings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* review: simplify the comment and drop the tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* review: ask for EP alone with an empty tp_plan
`enable_expert_parallel` without `ep_size` now sets `tp_plan={}`, and an
empty `tp_plan` applies no TP rule, so the choice survives a
`to_dict`/`from_dict` round-trip and has a spelling that outlives the flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* support quantization in dynamo/onnx and ET
* add qnn
* qnn
* add tests and docs
* tests
* style
* marker
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* comment
* et patch
* Address QNN review: single chipset authority, shared-graph calibration, docs
- Drop `config.soc_model`: the QnnQuantizer's chipset is the one authority
(annotations are chipset-specific); fp16 uses a code default.
- Calibrate the multi-token decode graph on the prefill captures too, so one
graph carries encodings for both distributions and the KV cache it writes
at prefill is read back under the same scales.
- `calibration_dataset` accepts any iterable (DataLoader batches included).
- Document the internal `_passes` wiring's exit plan (#20738 not in 1.4.1,
the public wrapper re-exports the module), QNN's multiple accelerators
(HTP today), and single-program deployment for generative exports.
- Re-verify the MoE/SSM quantization blockers on ExecuTorch 1.4.1: MoE's
int64 routing annotation still fails; Mamba moved from `CanonicalizeConv`
(fixed upstream) to `LiftConstantScalarOperands`. Skips updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* quantize moe
* Apply suggestion from @IlyasMoutawwakil
* drop qnn for now
* update docs
* Drop the QNN integration, keep the device-agnostic PT2E quantization and calibration
- Remove `prepare_for_qnn`, the QNN lowering branch and `_quantize` override, the QNN quantizer patch, the
`"qnn"` backend docs, and the QNN test cases.
- Revert the deferred patch-path resolution, which only existed for the QNN SDK's install-on-import.
- Say that each quantizer injects its own quantize/dequantize ops, which a backend may or may not support.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Use the require decorators and drop unnecessary guards
- `test_quantized_onnx` skips through `require_onnxscript` / `require_onnxruntime` instead of an inline check.
- `export_for_generation` reads `config.calibration_dataset` directly: every export config declares it.
- Import `torch.ao.quantization.fx._decomposed` unconditionally for the per-channel dequant translation; it
ships with every supported torch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Add backend-native quantizers for ONNX and OpenVINO
- `pt2e_quantizer` (renamed from `quantizer`) keeps the PT2E flow on the traced graph for every backend.
- `OnnxConfig.onnxruntime_quantizer` runs ONNX Runtime quantization (`OnnxRuntimeQuantizer`: static QDQ or
dynamic) on the converted model before it is saved; the ONNX exporter now saves after IR fixes.
- `OpenVINOConfig.nncf_quantizer` runs NNCF (`NNCFQuantizer`: `nncf.quantize` or `nncf.compress_weights`) on
the converted model, before state folding so calibration sees each sample's cache.
- OpenVINO keeps PT2E weights unfolded (`fold_quantize=False`), since it has no conversion for a lone dequantize.
- Add `is_nncf_available` / `require_nncf`; test all three model families on each path, gating OpenVINO int8
accuracy checks on VNNI.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Drop the OpenVINO `torch.empty_permuted` patch
Swapping it for a contiguous `torch.empty` changed the strides of torch's own decompositions that call it.
FakeTensor caches op output metadata process-wide, so those wrong strides outlived the patch: a later ONNX
export in the same process traced Mamba2's `reshape` as a `view` of a transposed tensor and failed. The op
already has an OpenVINO conversion extension (`_convert_empty_permuted`), as on the runtimes branch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Trim the quantization PR and stream calibration data
- Docs: the quantization example goes through `export_for_generation`, as the section advises; drop the repeated note.
- Calibration samples reach ONNX Runtime and NNCF one at a time instead of being materialized up front, and the ONNX
quantizer no longer copies the model.
- OpenVINO quantization tests derive the int8-activation case from the model's `FakeQuantize` nodes, return instead
of skipping when the CPU lacks VNNI, and use a 5% bar for weight-only compression (measured within ~2%).
- Shorter comments and docstrings; revert an import reflow in `exporters/__init__.py`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Tidy the ONNX Runtime and NNCF quantizers
- `OnnxRuntimeQuantizer` reads calibration feeds through a module-level `_FeedReader` and drops initializer
`value_info` only in dynamic mode, where ONNX Runtime needs it.
- `NNCFQuantizer` returns early for full quantization and passes `dataset=None` to data-free int8 modes.
- Name the helpers after what they run (`_run_onnxruntime_quantizer`, `_run_nncf_quantizer`).
- Stop passing save-only arguments to `torch.onnx.export`; the exporter's own `save` call takes them.
- `export_for_generation` folds the captured calibration into the per-component configs once.
- PT2E calibration keeps only the traced inputs of each sample in one copy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Move export quantization into an `exporters.quantizers` package
Quantizers become `ExportQuantizer` subclasses that declare the stage they run at (the `torch.export` FX graph or the
backend's own model), the export formats they support and the packages they need; the export validates them before
tracing. Each holds its own `calibration_dataset`, and `export_for_generation` hands every component a copy holding
that component's captured inputs.
- `PT2EQuantizer` (FX, every backend), `NNCFTorchFXQuantizer` (FX through NNCF's TorchFX backend),
`OnnxRuntimeQuantizer` (ONNX), `NNCFOnnxQuantizer` (ONNX) and `NNCFOpenVINOQuantizer` (OpenVINO).
- One `quantizer` config field replaces `pt2e_quantizer`, `onnxruntime_quantizer`, `nncf_quantizer` and
`calibration_dataset`.
- `DynamoExporter` traces under `torch.no_grad()`, so modeling `no_grad` regions don't leave
`wrap_with_set_grad_enabled` subgraphs that backends and NNCF can't lower; OpenVINO's own wrapper goes.
- Calibration warns only when a quantizer reads the sample-input fallback, and a sample missing the traced inputs
gets a clear error.
- Tests are named `test_<quantizer>_<backend>`, cover each quantizer on dense, MoE and SSM models, and run the
quantized ExecuTorch programs.
- Docs: an end-to-end example, every snippet verified, and the quantizer guide in "Extending the exporters".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Let quantizers build their own calibration set
`ExportQuantizer.quantize(model, sample_inputs, transform, export_format)` builds the `CalibrationSet` from the
quantizer's own `calibration_dataset` and calls `_quantize`, which each quantizer implements. Exporters only pass the
mapping from a forward-kwarg sample to their model's inputs, so `CalibrationSet` no longer appears in them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* trim
* Tighten the quantization docs and docstrings
Shorter wording for the generative-model, NNCF, calibration and per-component sections of the guide, the
`_quantize` description in "Extending the exporters", the NNCF and ONNX Runtime quantizer docstrings, and the
missing-traced-inputs error.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Address quantization review: configs, versioning, quantizer-owned calibration, >2GB ONNX
- Move the quantization docs to their own page (backend table, calibration, per-component
recipes, "Add a quantizer"); drop `.eval()` and `dict()` from the examples.
- Replace kwargs with `NNCFQuantizationConfig` / `NNCFWeightCompressionConfig` and
`OnnxRuntimeStaticConfig` / `OnnxRuntimeDynamicConfig` (`to_kwargs()` passes the set fields).
- Version the quantizers like the exporters: shared `check_packages`, `required_packages`
listing everything each quantizer imports, `tested_versions` and `min_versions`.
- Drop `QuantizationStage`: a quantizer declares `quantizes_fx_graph` and implements
`quantize(model, sample_inputs, export_format)`, building its own calibration feed;
the exporters just call it.
- Accept any mapping as a calibration sample (e.g. `BatchEncoding`).
- Re-export the FX-quantized graph non-strict (strict re-export of a GraphModule fails).
- ONNX quantizers take and return `onnx_ir.Model` and quantize through files with external
data, so models past protobuf's 2GB limit work.
- `NNCFOnnxQuantizer` folds initializer `Transpose`s so NNCF quantizes a tied `lm_head`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Fix the export_for_generation anchor in the quantization docs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Show XNNPACKQuantizer for ExecuTorch in the PT2E docs; drop the extend-page pointer
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Move the calibration helpers to quantizers.utils; import like the exporters
- `capture_calibration_inputs`, `traced_inputs`, `onnx_feed` and `openvino_feed` move from
`exporters/utils.py` to `exporters/quantizers/utils.py`, next to the quantizers that use them.
- The quantizers import their libraries at module level behind `is_*_available()`, like the
exporter modules, instead of inside `quantize`.
- Drop the runtime availability checks inside `TYPE_CHECKING` blocks.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Import the exporters lazily
`transformers.exporters` and `transformers.exporters.quantizers` become `_LazyModule`s, like
`transformers.integrations`, and `AUTO_EXPORTER_MAPPING` holds the built-in exporters as module
paths imported on first lookup. `import transformers.exporters` drops from ~9.4s to ~0.8s, and
an exporter only loads its own backend (`DynamoExporter` no longer pulls in ExecuTorch,
OpenVINO or onnxscript).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Update docs/source/en/exporters_quantization.md
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Review: tested versions for every quantizer dependency, one package error, per-backend test classes
- Each quantizer lists a tested version for every package it requires (NNCF's subclasses extend
`_NNCFQuantizer`'s), not only its own library.
- `check_packages` reports missing and outdated packages together, in one `ImportError`.
- NNCF quantization defaults `model_type` to TRANSFORMER with `setdefault`, so a config's own value wins.
- The NNCF docs example lists its configs side by side before the export.
- The quantization tests split into Dynamo/ONNX/OpenVINO/ExecuTorch classes over a shared
`QuantizationTesterMixin` (memory cleanup, seeded once in `setUp`); `_tiny_model` is `_dense_model`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Quantization tests: one PT2E helper returning a PT2EQuantizer, like the NNCF one
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Add Kolibri 1
* fix(kolibri1): fall back to the fp32 router upcast when dtypes differ
`torch.mm(..., out_dtype=)` rejects operands of different dtypes, e.g.
fp32 activations into a bf16 router under autocast at inference.
* fix(kolibri1): reject configs without a sliding window
Without one, the attention inherited from EXAONE 4 applies RoPE in every
layer, a model Kolibri 1 was never trained as.
* fix(kolibri1): drop the softmax load-balancing loss
Kolibri 1 balances experts through `e_score_correction_bias`, which
training updates outside the loss. The loss inherited from Qwen3-MoE
ranks experts by softmax over unbiased logits, so it balanced a routing
the model does not use. Other bias-routed models (DeepSeek-V3, GLM-4-MoE,
EXAONE-MoE) have no aux loss either.
* docs(kolibri1): use a Kolibri checkpoint in the causal LM example
* docs(kolibri1): trim the expert-parallel plan comment
* test(kolibri1): add integration tests on the BF16 checkpoint
* fix(kolibri1): update model name and contribution date
* docs(kolibri1): note that the CUDA router fast path is for performance
---------
Co-authored-by: Sascha Wirges <sascha.wirges@aleph-alpha-research.com>
Co-authored-by: Eric Schreiber <eric.schreiber@aleph-alpha-research.com>
Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co>
Arthur