A compressed-tensors checkpoint that only quantizes the KV cache
(`kv_cache_scheme` set, no `config_groups`) crashed at load time:
`get_experts_scheme` indexed `groups[0]` on an empty list from
`serialized_names` and `get_load_operations`. Guard both on empty
config groups, like the existing sparsity-only case, and add a CI
test on inference-optimization/TinyQwen3-KV-Cache-Only that checks
the scales load onto every attention module.
* Match a skip-list entry from the start only up to a module boundary
should_convert_module matched an entry from the start of the module name with
no boundary after it, so "model.layers.1" also skipped model.layers.10 to 19,
and Qwen3.8-27B-FP8's router entry "mlp.gate" also skipped the quantized
mlp.gate_proj (its FP8 weight then reached a bf16 linear). A match from the
start now has to end on a "." it consumed, or right before a "." or the end
of the name, which keeps regex entries that carry their own boundary such as
"^model\.audio_tower(?:\.|$)". The endswith clause Hub skip-lists rely on
("gate" covering shared_expert_gate) is unchanged.
Supersedes #47990, which anchors with "$" and so stops matching such regex
entries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Shorten the should_convert_module docstring
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Restore the matching-rules comment of should_convert_module
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Move the should_convert_module test to a general quantizer utils test file
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Explain why the skip-list key is grouped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Fit the skip-list comment within the noisy-comment limits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Name the skip-list test's exclusion lists modules_to_not_convert
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Since DeepSeek-V4 declares `ep_dispatch_experts`, the Mega MoE override left that rule in place: token dispatch
wrapped the Mega MoE experts and called them without a process group, so `tp_size=ep_size` runs raised on the first
forward. The older `ep_router` + `moe_tp_experts` plan was never applied either, since the override renamed both
styles and the mixin matched neither.
Mega MoE dispatches inside its kernel, so it takes over the experts rule (`ep_dispatch_experts` ->
`megamoe_experts`), runs on the TP group like masked EP, and requires `ep_size == tp_size`.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* first cleanup
* more cleanup
The various agent PRs most probably did not understand the core logic /
idea behind the weight converter, this left a lot of method patching
here and ther.
1. Deserialize: some method just need a cast, other need more work. The
ConversionOps added as the "first" operation, updating the weights
conversion (each one of them) are here for that.
2. TP: bloated code to just update a plan.
3. Patch: I have no idea how, but we accepted plans that had explicit
overwrite. This is legacy
* update
* sub config fixes
Saw another PR recently: #48759, had the same bug. this adresses it
* more simplification
* debloat more shit, standardize
* nits
* fucking bloatland
* Fix PR slow CI suggestion install: non-editable install
* update now that the $ bloat is gone
* Fix PR slow CI suggestion: install transformers in a venv (system site-packages not writable)
* metal tests: unanchored weight key
* nits this time
* up
* remove last pieces of dead code
* fix failing test
* fix test
* pr comments
* human coments
* Treat scales and etc individually?
* use a map
* replicate explicitely
* impl copy
* Quantized element size as property, one-line FP8Experts comments, ue8m0 float32 scale loading test
* Drop the $ anchor from the bailing_hybrid expert_bias renaming, patterns end on a name boundary now
* Restore the 1 byte quantized element size of FineGrainedFP8
* Quantized element size as a property in every quantizer
- Expects the same greenlist token in the watermark test on XPU and other devices.
- Uses the multi-accelerator requirement for GPTQ device-map tests instead of a multi-GPU-only requirement.
- Runs the MXFP4 low CUDA compute-capability tests only on CUDA, since their mocked CUDA capability is not read on XPU.
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
* GGUF tokenizer: keep the file's user-defined tokens whole
A GGUF marks tokens like `<think>`, `</think>`, `<tool_call>` and `<tool_response>` as
USER_DEFINED. llama.cpp always matches these whole in the text, and the reference tokenizers
carry them as non-special added tokens. The GGUF-built tokenizer only registered CONTROL
tokens, so `<think>` was cut into `<`, `think`, `>`: every chat prompt for Qwen3.5 differed
from what llama.cpp feeds the model. Add them as plain added tokens, restricted to spellings
already in the vocabulary so a sentencepiece file's raw whitespace runs do not grow it.
* update comment
* support Qwen3.5-MoE GGUF files
Adds the `qwen35moe` architecture to the config, conversion and tokenizer mappings, the expert-bank
tensors to the loading path, and grafts the router and zero-centred norm kernels onto the model. The
layers are named `SoftmaxTopKRouter` and `RMSNormZeroCentered` rather than `TopKRouter`/`RMSNorm`: a
kernel for the plain names would reorder softmax and top-k, exact only for a softmax router, and
would compute `x * w` where this norm needs `x * (1 + w)`.
* clean
* cleaned
* better !
* update kernels
* up
* update
* initial draft
* Address serge review: audit the no-grad default and pin the private hook
* You are such a verbose agent.
* Review round from @ydshieh, @vasqu and @stevhliu
* vasqu round 2: docs cross-reference, a note block, a class name, and test_kernels
* @ydshieh round 2: rename the grad knob, name the numbers, explain the rest
* tweak the kernels dep
* Skip gpt_oss kernels-dependent tests without kernels
* OOM guards
* Make the CPU-RAM probe container-aware and guard host RAM in gpt-oss
`psutil.virtual_memory().total` reads the host `/proc/meminfo`, so inside a K8S pod it reports the
whole node (~750 GiB) and says nothing about the cgroup limit the OOM killer enforces. The
`CI_CPU_MEMORY_LIMIT_GB` patch in `conftest.py` only half-covers this: it deliberately caps psutil at
`limit * num_accelerators` (a `device_map="auto"` planning budget), which is 120 GiB on a 2-GPU runner
whose cgroup still kills at 60.
- add `get_cgroup_memory_limit_bytes()` (cgroup v2 `memory.max`, v1 `memory.limit_in_bytes`, with
their respective unlimited sentinels)
- add `get_cpu_ram_total_gib()`, taking the min of the cgroup limit, psutil, and the raw
`CI_CPU_MEMORY_LIMIT_GB`
- rebuild `require_large_cpu_ram` on it, and accept `@require_large_cpu_ram(memory=N)` as well as the
bare form, matching `require_torch_large_accelerator`. This also fixes its two existing users
(gemma2, longcat_flash), which were subject to the same host-vs-pod bug.
- guard `test_model_outputs_distributed` on host RAM too: `distributed_worker` loads with no
`device_map` and moves the model afterwards, so the whole checkpoint transits host RAM first --
that is the allocation that got the CI container SIGKILLed.
Also cite the exact weight footprints from `modeling_utils.get_total_byte_count` on a meta-device
model (38.96 GiB for 20b, 217.61 GiB for 120b) so the budgets are traceable rather than guessed.
* Address review: separate the measured CPU RAM limit from the CI budget
Follow-up to @ydshieh's review on #48118.
`get_cpu_ram_total_gib` took the min of three values, one of which was `CI_CPU_MEMORY_LIMIT_GB` read raw. That
variable is a *per-accelerator* budget, so on a 2-accelerator A10 runner it reads 60 GiB where the runner really
has 180 -- every guard on that runner would over-skip. Multiplying by the accelerator count (what `conftest.py`
already does when patching psutil) gets to 120, still not 180, because the budget is an allocation policy for
`device_map="auto"` rather than a measurement of the machine.
So split the two apart:
- prefer what can be measured -- the cgroup limit the OOM killer enforces, and the machine's physical RAM --
taking the smaller, since each is wrong on its own (no cgroup outside a container; physical RAM reports the
whole node inside a pod)
- fall back to the CI budget only when neither can answer, i.e. on an instance-sharing runner with no cgroup
limit of its own
- add `get_ci_cpu_memory_budget_gib` as the single source of truth for the per-accelerator arithmetic, and have
`conftest.py` use it instead of its own inline copy
- add `get_physical_cpu_ram_gib`, which reads past the `patch_psutil_cpu_memory` cap. Without this the fallback
above is unreachable in CI: conftest patches psutil at session start, so "physical RAM" would report the
planning budget back to us rather than the machine.
Return `inf` rather than 0 when nothing can answer at all -- no cgroup, no psutil, no CI budget. That is an
ordinary local setup, not a broken one, and 0 silently skipped every memory-guarded test. `require_large_cpu_ram`
loses its `is_psutil_available` early-return, now redundant; this also closes a gap where a container *without*
psutil ran the test blind instead of consulting its cgroup limit.
Also note in the docstrings that the 80 GiB default is "more than our 60 GiB GPU runners" rather than an estimate
of any model, and that ROCm reaches `get_accelerator_total_memory_gib` through the cuda branch (a HIP build
reports `torch_device == "cuda"`), since the existing comment read as though it were excluded.
Tests cover the resolution order in all four environments, including that the guard reads past the psutil cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* add distributed config
* Add native FSDP2 module and migrate FSDP imports (Phase A PR-2).
Move FSDP2 wrapping and plan verification to distributed/fsdp.py, keep
integrations/fsdp.py as a backward-compatible re-export, and update core
call sites to import from transformers.distributed.fsdp.
* linting
* unecessary
* copyright edit
* revert
* add shard on read
* jsut shard on read
* cleaning
* linting
* fix
* fix
* remove redundant test file
* Update src/transformers/distributed/fsdp.py
naming
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* avoid looping, just look at dict
* expand_fsdp returns reshard_targets, no_reshard_targets right away
* better _resolve_tied_embed_lm_head_plan
* cleaning
* ruff
* more robust detection of embed and lm_head
* cleaning
* ruff
* typo
* cleaner
* cleaner
* typo
* refactor dense path + apply_contiguous_shard
* linting
* cleaning
* refactor _apply_strided_shard
* better
* refactor _slice_and_cat
* better comment
* refactor moe dtensor shard ops
* better comment
* comment
* cleaning
* linting
* Add FSDP orchestration: mesh init, distribute-before-load, and DCP save.
Wire distributed_config from_pretrained/save_pretrained alongside the legacy tp_plan path, add distributed/utils.py for mesh orchestration and checkpoint I/O, and extend sharding_utils with DTensor gather/optimizer fusion helpers needed by save/load.
* add fsdp plan to 2 models for now
* add tests fsdp mixin
* linting
* refactor test fsdp mixin
* test fsdp mixin cleaning
* remove fsdp policy in tests + trim down further
* test fsdp clean
* restore test_modeling_utils
* linting
* start trim down stuff
* fix
* breaking: cleaning modeling_utils.py
* load path with fsdp (dtensor) and tp (old tp) is linked
* linting
* add saving
* styling
* fix tp ci
* add fsdp to ci
* linting
* pick one model only for this PR
* restore
* trigger fsdp ci
* doc cleaning + tp_size remove
* fix tp ci for ep
* edit doc
* move distributed function to utils + guarding
* linting
* expand_fsdp_plan iterate over modules
* comment about tie embedding
* add comment tied embedding
* add DistributedMixin
* some cleaning
* cleaning + comment
* rename function for clarity
* Apply suggestion from @ArthurZucker
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* doc
* comment
* linting
* refactor
* abstract to mixin
* typo
* Add FSDP plans to all models from distributed branch.
Port base_model_fsdp_plan and ForCausalLM _fsdp_plan entries from PR #46269
and expand FSDP distributed test coverage to the pilot model subset.
* fsdp plans
* linting
* linting
* Add distributed runtime utils and DistributedMixin (FSDP orchestration 1/3).
Introduce distributed/utils.py and DistributedMixin, defer DistributedConfig
validation to load time, and refactor PreTrainedModel plan properties without
changing the from_pretrained distributed_config API yet.
* Wire DistributedConfig through from_pretrained and save_pretrained (FSDP orchestration 2/3).
Route distributed loading and saving through DistributedMixin, migrate TP tests
and docs off tp_plan="auto", and add FSDP gather/DCP save paths.
* Add FSDP CI and end-to-end FSDP tests (FSDP orchestration 3/3).
Add FSDPTesterMixin, cohere2_moe base_fsdp_plan, dedicated fsdp_ci job, and
pytest markers for distributed FSDP load/save/generation coverage.
* addd ep_plan
* restore validate module
* Wire DistributedConfig through from_pretrained and save_pretrained.
Route distributed load/save orchestration through DistributedMixin so TP and FSDP paths share the same entry points.
* revert
* inline distribute_model
* revert
* remove saving/loading
* leaner mixin
* downgrade torch version guarding
* remove
* linting
* revert
* revert
* post_init() parallel plan move to mixin
* revert tp mixin
* add save/load
* only FSDP save/load for now
* revert
* refactor
* modular
* ea
* begin migration TP
* clean up
* add it to pretrained model
* linting
* fix the test by moving in init class the fsdp plan instead of post init
* edit
* replace everything
* begin migration TP
* clean up
* revert merge conflict
* local params for forward
* cleaning
* migration from integration.tensor_parallel to distributed.tensor_parallel
* revert models
* revert
* fix
* breaking: cleaner way to_local for quantize weight(almost done need to fix backward)
* fix all tests
* TP dtensor handle natively the to_local() for deepgem + fp8 (#47634)
* Refactor distributed tensor_parallel module and TP mixin tests
* better approach to to_local
* pass as class method
* cleaning PackedColwise
* remove the use of to_local in kernels to offload this task to TensorParallel
* rowwise bias after redistribute
* make it easier to understand
* requires grad only for floating point numbers
* fix tp meagamoe fp8 with dtensor
* linting
* revert gitignore
* fix run slow quantization ci
* fix deepseek v4 ep backward tests
* fix deepseek glm4 moe tp backward pass
* Refactor tensor parallel loading logic by removing unused tp_plan handling and updating is_dtensor type hint for better type safety.
* linting
* bench: dtensor vs legacy TP
* bench: dtensor vs legacy TP (#47728)
* bench: dtensor vs legacy TP
* clean rowise
* refactor colwise
* comment
* comment
* refactor MoEExperParallel
* ruff
* add rowwise input
* remove helper script
* remove dead code in mxfp4
* remove old TP
* cleaning
* cleaning
* claening
* add test_shard_tensor_shape_consistency
* cleaning
* small fix
* typo
* comment
* rename colwise_gather_output to colwise_rep
* keep tp import to avoid breaking chnges
* remove async_op=True for redistribute
* linting
* renaming MoeExpertsParallel class
* ep router doc
* mlinter
* add todo
* fix
* remove unused keys from global_mapping to avoid BC
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
torchao removed the torchao/dtypes package (AffineQuantizedTensor and the
Layout classes). This updates the transformers torchao integration to stop
importing from torchao.dtypes so it no longer crashes against current
torchao, mirroring the approach in transformers #45321.
Changes:
- integrations/torchao.py: drop the `_quantization_type` helper and its
`from torchao.dtypes import AffineQuantizedTensor` isinstance branch;
`_linear_extra_repr` now detects quantized weights inline via
`isinstance(self.weight, TorchAOBaseTensor)` (from torchao.utils).
- tests/quantization/torchao_integration/test_torchao.py: replace the
`from torchao.dtypes import AffineQuantizedTensor` import with
`from torchao.utils import TorchAOBaseTensor` and update the isinstance
assertions accordingly.
- docs/source/en/quantization/torchao.md: remove the MarlinSparseLayout
("int4-weight-only-24sparse") examples and migrate the Int4CPULayout
example to `Int4WeightOnlyConfig(int4_packing_format=PLAIN_INT32)`.
Test plan:
- `python -c "import transformers.integrations.torchao"` imports cleanly.
- `RUN_SLOW=1 pytest tests/quantization/torchao_integration/test_torchao.py`:
all 18 TorchAOBaseTensor isinstance / FqnToConfig tests pass. The 3
remaining failures (test_int4wo_quant, test_int4wo_offload,
test_serialization_accelerator_*_Int4WeightOnlyConfig) are pre-existing
generation-output text comparisons pinned to torchao int4 kernel numerics
on the test hardware, unrelated to this change.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Summary:
1. fix torchao NVFP4 serialization with transformers
2. add a test to cover the fix
While i'm here, also did the following bundled into this PR:
3. make the torchao serialization test have human readable names (easier
to debug)
4. fix the float8 test (update the expected output)
after this PR the test command for all torchao configs passes on an
NVIDIA B200
Test Plan:
```
RUN_SLOW=1 pytest tests/quantization/torchao_integration/test_torchao.py -k "Serialization" -s
```
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* Add GPT-OSS GGUF support with YaRN rope scaling reconstruction
* Add GGUF loading test suite for GPT‑OSS
* docs: add GGUF loading section to gpt_oss.md
* fix: correct import of GptOssTensorProcessor in test; remove from model __all__
* Finalize GPT‑OSS GGUF support: move test, adjust config reconstruction
* fixed docs not closing example bracket
* Fix lint: remove trailing whitespace
* Fix tensor construction consistency
* reverting to original docs
* deep gemm
* standardize
* clear deepgemm flow and blackwell optimization
* avoid unnecessary gathers in grouped mm
* assertions and drop the synced path
* use lazy load kernel
* style
* add prefix and check for cuda runtime version
* better names
* exit on missing functions
* comment about why we use deepspeed cutlass single gemm
* global statements
* add cuda check
* fix deepgemm fastpath for models with bf16 scales
* force fp32 scales in experts as well
* Update src/transformers/integrations/finegrained_fp8.py
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Update src/transformers/integrations/hub_kernels.py
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* fix
* style
---------
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* fix: split MXFP4 dependency checks for specific error messages
Fixes#44912. The combined `kernels_available` check made it impossible
to identify which dependency (Triton vs kernels) was missing. Now each
check has its own actionable message with install instructions, following
the established pattern in quantizer_higgs.py.
Also hardens is_triton_available() and is_kernels_available() against
"N/A" version strings from _is_package_available().
* address review: revert import_utils.py changes, reduce tests to 2
* sinq integration files
* sinq integration update
* sinq integration no lazy import
* Tests for sinq integration
* minor changes to sinq integration
* sinq integration documentation added
* small correction to sinq documentation
* small correction to sinq documentation
* remove auto_patch_io flag and fix the selection of the device for sinq integration
* remove auto_patch_io flag and fix the selection of the device for sinq integration
* remove auto_patch_io flag and fix the selection of the device for sinq integration
* remove auto_patch_io flag and fix the selection of the device for sinq integration
* Code style fix sinq integration
* minor changes in comments for sinq integration
* add for documentation sinq integration
* add documentation for sinq integration
* minor adjustment in sinq quantizer
* minor changes to sinq integration
* delete debugging print in sinq integration
* sinq integration files
* sinq integration update
* sinq integration no lazy import
* Tests for sinq integration
* minor changes to sinq integration
* sinq integration documentation added
* small correction to sinq documentation
* small correction to sinq documentation
* remove auto_patch_io flag and fix the selection of the device for sinq integration
* remove auto_patch_io flag and fix the selection of the device for sinq integration
* remove auto_patch_io flag and fix the selection of the device for sinq integration
* remove auto_patch_io flag and fix the selection of the device for sinq integration
* Code style fix sinq integration
* minor changes in comments for sinq integration
* add for documentation sinq integration
* add documentation for sinq integration
* minor adjustment in sinq quantizer
* minor changes to sinq integration
* delete debugging print in sinq integration
* Adapt sinq integration to transformers v5
* sinq integration for transformers v5
* Added part of the suggested modifications to make the code simpler
* Modification of the quantization flow and remove of asinq option
* Minor adjustments and creation of fuction to substitute quantized layers
* Eliminate device specification in SinqConfig and tests script adaptation
* final adjustments and checks
* final checks
* Fix merge conflict in import_utils
* Fix code quality
* update tests and fixing minor typos
* Fix grammar of warning message
* Update tests/quantization/sinq/test_sinq.py
* Apply repo consistency fixes
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
Co-authored-by: Mohamed Mekkouri <93391238+MekkCyber@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Kyle Sayers