mirror of
https://github.com/huggingface/transformers.git
synced 2026-10-29 01:06:24 +08:00
fd798ba130
* Assert cached decode matches recomputing without a cache `GenerationTesterMixin` already ran greedy generate with and without a cache, but each test only checked its own shapes — nothing compared them, so a cache that feeds its layers the wrong positions or a mask of the wrong width passed both. Driving each side through `generate` keeps the modality, encoder-decoder and input-naming plumbing in the library, which is what lets one assertion cover text, VLM and encoder-decoder models alike. It compares per-step logits, then ids up to the first near-tie — tie-checking the scores rather than the logits, since the ids are picked after the processors run and `min_new_tokens` drives EOS to -inf. Models whose state *is* their cache skip on `_is_stateful`. Across every model it found four real bugs: - fsmt embedded positions from the sliced one-token input, so every decode step re-read position `padding_idx + 1`. On real weights the drift flips a token by step 6; an earlier commit had overwritten the expected en-ru translation to absorb it, and the fix restores the original text for all four pairs. - axk2, deepseek_v32 and glm_moe_dsa fold their sparse top-k into the attention mask with `masked_fill`, which writes True — i.e. *attend* — when the mask is boolean, as it is on the sdpa path. Padded sequences longer than `index_topk` attended future tokens. - paligemma (and v1) let generated tokens inherit the prompt's `token_type_ids`, putting them in the bidirectionally-attending prefix block, so without a cache earlier tokens attended tokens generated after them. - vibevoice_asr dropped its audio inputs after prefill even with no cache, so the placeholders fell back to text embeddings. blt (decode rebuilds hash n-grams and patch lengths from one byte and gives the global trunk no cache), vibevoice_asr's residual VAE sampling and prophetnet's collapsed single-step branch are skipped with the measured mechanism. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * style * Address review: shared assertion, modular non-inherit, shorter comments Use `assert_similar_generate_outputs` as both reviewers asked, and keep a logits comparison alongside it at a looser 1e-2: the helper returns as soon as the token ids match, so on its own it cannot see a cache bug that moves the logits without flipping the argmax — measured at 0.0187 for axk2's indexer, which the ids agreed through. Stop the four gemma variants from inheriting PaliGemma's `token_type_ids` update by stubbing the method with `raise AttributeError(...)` in the modular source, which drops it from the generated files entirely so they take `GenerationMixin`'s. The converter matches `raise` on the line right after the `def`, so the reason goes in the exception message rather than a comment. Shorten the paligemma and fsmt comments, drop the vibevoice_asr one, and guard git's mask widening on a 2D mask. Also revert `ModelOutput` import churn in inkling, kimi_k25 and muse_glimmer that was unrelated to this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Drop the sparse-indexer mask change, to be sent as its own PR The additive-mask fix for the indexer path (axk2 / deepseek_v32 / glm_moe_dsa) is independent of the cached-decode parity work and is easier to review on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Keep FSMT's positional-embedding signature BC-safe `SinusoidalPositionalEmbedding.forward` used to take `(input, incremental_state, timestep)`; neither extra argument was ever read, so they were dropped for `past_key_values_length`. Passing the new one positionally would silently reinterpret an old caller's second argument as a cache length, so it is keyword-only: such a call now raises instead of returning wrong positions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Drop the signature note from FSMT's positional embedding The `*` speaks for itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Group the non-inherited generation hook with the other generation methods `_update_model_kwargs_for_generation` sat above `forward`; it belongs next to `prepare_inputs_for_generation` / `create_masks_for_generate`. Generated files are unchanged — the stub's only effect is to keep PaliGemma's override out of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * skip for now * corrected model --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> Co-authored-by: vasqu <antonprogamer@gmail.com>