mirror of
https://github.com/huggingface/transformers.git
synced 2026-10-29 01:06:24 +08:00
Tarek ZiadeandClaude Opus 5
ba356d2b33
* future mlinter 0.1.5 * fix rule 58 * Add tokenizer tests for TRF038 Adds the missing tokenizer test files that mlinter's TRF038 reports, now that the rule covers tokenization_*.py files. * fix roberta * Allowlist the three alias-only tokenizer modules for TRF038 bart, mobilebert and squeezebert started firing when mlinter 0.1.5 extended TRF038 to tokenizer files. Their tokenization_*.py define no classes at all, each being a two-line back-compat alias left by "rm slow tokenizers" (#40936), which deleted their test files on purpose. A test file here could only assert alias identity, and the aliased tokenizers are already covered by tests/models/roberta and tests/models/bert. Allowlisted as a placeholder: the better fix is upstream, where TRF038 should skip a source file that defines no classes. TRF038 now reports no findings. * Route composite sub-configs through CONFIG_MAPPING for TRF009 mlinter 0.1.5 extended TRF009 from modeling files to every model file, which surfaced six configs importing a sibling model's config class directly. Each now resolves it through CONFIG_MAPPING with AutoConfig as the sub_configs marker, which is what the rule prescribes and what 99 other composite configs already do: cohere_asr ParakeetEncoderConfig -> parakeet_encoder esmfold2 EsmcConfig -> esmc gemma4_assistant Gemma4TextConfig -> gemma4_text gemma4_unified_assistant Gemma4UnifiedTextConfig -> gemma4_unified_text ovis2 Qwen2Config -> qwen2 sam3 CLIPTextConfig -> clip_text_model cohere_asr already used CONFIG_MAPPING for instantiation, so only its sub_configs value and import changed. Where the imported class also appeared in a type annotation it is widened to PreTrainedConfig, and esmfold2's _init_nested helper takes the class from CONFIG_MAPPING since it calls cls() directly. Verified: default construction and to_dict/from_dict round-trip resolve the same concrete config class as before for all six, and tests/models for the six models pass (552 passed, 551 skipped). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Backport mlinter rule-prose compression and allowlist the rest of TRF009 Two changes to utils/rules.toml. Backports transformers-mlinter d0ac218 ("compress the rule explanation prose", before: upstream's file as the base, with this repo's licence header and its allowlist_models entries and their justification comments re-applied on top, so the only remaining difference from the installed mlinter/rules.toml is the header and those allowlists. make fix-repo regenerated docs/source/en/modeling_rules.md. Allowlists the ten TRF009 findings left after the CONFIG_MAPPING fixes, in three groups, each with its reasoning recorded in the file: - Back-compat tokenizer shims (bart, convbert, distilbert, fnet, mobilebert, squeezebert): the cross-model import is the module's whole purpose. - generation_*.py (nemotron3_5_asr, nemotron_asr_streaming): the rule points at modular files, but the converter has no generation_*.py support, so the prescribed remedy does not exist yet. - sam3: CLIPTextModelWithProjection is needed for checkpoint key compatibility. shieldgemma2: fixable with a modular file, left as follow-up. make typing and make check-repo now pass. * Fix vibevoice for mlinter 0.1.5 after rebase VibeVoice (#40546) landed on main after this branch forked, so it was never checked against the 0.1.5 rules and brought in two findings. TRF058: VibeVoiceDiffusionHeadSinusoidalEmbedding created `freq` via `self.register_buffer(...)`, which cannot be inherited and tweaked from a modular file. Assign `nn.Buffer(..., persistent=False)` instead, which also makes the `freq: torch.Tensor` linting workaround unnecessary. Same non-persistent buffer and same forward output. TRF009: allowlist `vibevoice`. generation_vibevoice.py imports a runtime cache class from its own sub-model package; the family already routes its sub-configs through CONFIG_MAPPING, and a cache class cannot come from CONFIG_MAPPING. It also sits in a generation_*.py file, which the modular converter does not support, the same blocker as the nemotron entries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Adopt transformers-mlinter main: sync rule prose, drop dead allowlist entries Re-ran the checker against transformers-mlinter git main (e6533da). Green, and the bump makes a chunk of this branch's curation obsolete. Synced the rule prose the repo copy mirrors, which had gone stale because the described behavior changed upstream: TRF029 -- an optional `None`-default parameter is now an override, not a second source of truth, so it is exempt TRF035 -- F401/F821/F822 are now accepted in modular files TRF041 -- guards (an `if` with no `else` that only raises or only logs) and a `DEFAULT_EXEMPT_ATTRIBUTES` list of framework plumbing are exempt Dropped 39 allowlist entries that no longer fire, mostly mlinter#27 (the rule engine now resolves a class's base chain across models instead of only inside the linted file), which this branch had flagged upstream rather than worked around: TRF018 16 -> 1 only `radio`, the one real case, is left TRF020 1 -> 0 the `axk2` false positive is gone TRF034 17 -> 5 the 12 cross-file false positives are gone; the 5 real findings stay as follow-ups TRF029 2 -> 1 TRF035 5 -> 0 TRF041 14 -> 9 Each rule's comment block now describes what is left and why. Verified minimal: emptying any of these six allowlists reproduces exactly the kept entries. 24 further entries (TRF002, TRF004, TRF005, TRF006, TRF009 `maskformer`, TRF010, TRF019, TRF056) are also dead, but they were already dead before this bump, so they are left alone here. TRF037's `x_clip` stays: the rule is opt-in, so it never fires and the entry documents what would. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Regenerate modeling_rules.md for the synced rule prose Generated from utils/rules.toml by utils/check_modeling_rules_doc.py; picks up the TRF029, TRF035 and TRF041 prose synced from transformers-mlinter main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop 19 more allowlist entries that no longer fire Swept every rule the way the earlier mlinter#27 pass was done: emptied each allowlist, re-ran, and checked what actually fires. 213 entries -> 194. "Does not fire" has four causes, so each candidate was classified rather than deleted on the strength of a green run: - the model is compliant now - the pattern survives only in a generated file, copied from a parent - the model is grandfathered by cutoff_date, so it is never checked - the rule is opt-in and never runs at all Only the first two are safe to drop. To separate them, each candidate rule was run directly against the model's own source with grandfathering off. Removed as compliant, verified rule-by-rule against the source (14): TRF005 9 -> 0 the rule accepts an explicit `_no_split_modules = None`, a modular `AttributeError` sentinel, and raw-string elements TRF010 2 -> 0 both configs carry @strict(accept_kwargs=True) TRF006 4 -> 3 TRF009 15 -> 14 (maskformer) TRF056 2 -> 1 (kimi_k25) Removed as generated-only copies (5): TRF004 9 -> 4 hubert, sew, unispeech, unispeech_sat and wavlm define no `tie_weights` in their modular files; the override exists only in their generated modeling_*.py, copied from Wav2Vec2ForCTC. wav2vec2 still fires, so the root cause stays tracked -- listing all five implied five separate bugs. Kept deliberately, and now documented in the file (5): TRF002 lighton_ocr -- `base_model_prefix = ""` is real and authored in its own modular file; the rule misses it only because it resolves base chains within a single file, so `LightOnOcrModel(Mistral3Model)` is not recognized as a pretrained class TRF019 blip_2, grounding_dino, omdet_turbo -- grandfathered by cutoff_date; they still set `_defaults` and fire once grandfathering is off TRF037 x_clip -- the rule is opt-in, so the entry records what would fire Verified: emptying every allowlist now reproduces exactly the kept entries, except those 5, each of which has a stated reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Adopt the TRF034 trunk scoping from mlinter main mlinter#65 scoped TRF034 to the model's checkpointable token-mixing trunk. Sync the rule prose and cut the allowlist from 5 models to 1. `dinov3_convnext`, `openai` and `xcodec2` never set `supports_gradient_checkpointing`, so `gradient_checkpointing_enable()` raises for them rather than skipping a layer and there was nothing to report. `tipsv2_dpt`'s three entries are DPT head layers, not a token-mixing stack. `x_clip`'s `PromptGeneratorLayer` stays: an attention stack on a model that does enable checkpointing, so it is a real finding and a follow-up for that model. Over this checkout with every cutoff neutralised, TRF034 goes from 103 findings over 76 models to 20 over 19, and every survivor is a transformer stack in a checkpointable model. * Pin transformers-mlinter 0.1.5 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>