mirror of
https://github.com/huggingface/transformers.git
synced 2026-10-29 01:06:24 +08:00
Arthur
b7e0c20b9f
This reverts commit 1b20eb168c.
#47488 replaced the `Metaspace` pre-tokenizer with a `Prepend("▁") + Replace(" ", "▁")`
normalizer. That changes tokenization in two ways that break existing checkpoints.
1. Added-token contents are themselves normalized when the added-vocabulary matcher
(daachorse) is rebuilt, so `<s>` is stored as `▁<s>` and stops matching `<s>` in the
text. Any checkpoint shipping its specials with `normalized=True` loses them:
tok.tokenize("Hey how are<s>you?<unk> <unk>")
['▁Hey', '▁how', '▁are', '<', 's', '>', 'you', '?', '<', 'unk', '>', '▁<unk>']
2. The prepend runs on every chunk produced by the added-token split, so a metaspace is
inserted after every special token. `[INST] How are you doing?<s>[/INST]` moves `[`
from 29961 to 518, i.e. every multi-turn Llama-2 chat prompt tokenizes differently.
#47488 wrote this into its own expected outputs rather than treating it as a bug.
The leading-whitespace round-trip reported in #47487 is real, but it is decode-only and
does not need a pipeline change.