* fixes missed
* gemma test fix
* refactor
* rm legacy from llama
* added renaming
* add _model
* update legacy
* update legacy
* fix docstring
* always load blank, then set _tokenizer if we have it
* new toks
* update all berttokenizer based models
* apply feedback - delete bert duplicates
* more models --> fast only
* more convert_slow models
* fix common test refs
* updating fast only tokenizers
* openai and pegasus
* enable sentencepiecebackend
* more models
* code gen
* t5
* code gen tests
* speecht5
* mbart
* mbart50
* more models
* more models
* layouglmv2
* update tests
* update tests
* update tests
* pretrainedtokenizer
* whisper
* whisper
* layoutxlm and storing backends
* refactor sentencepiecebackend and additional_special_tokens
* renaming tokenization_utils --> tokenization_python
* udpate tests
* bert test
* blenderbot
* clip
* codegen
* code_llama
* cohere
* deberata, deberat v2, funnel
* gpt2
* batch update tests
* pegasus qwen2 roberta
* more models
* layout tests
* some renaming
* fix references to utils_fast
* fix refs
* fix refs
* fix refs
* fix refs
* fix refs
* fix refs
* fix refs
* fix some tests
* regression
* fix refs
* fix refs
* missed the most crucial file in my last commit
* fix refs
* fix refs
* fix refs
* batch encode fix
* fix some tests
* BC for batch_decode bc too many refs
* more tests
* fix more tests
* fix for processors
* fixing more models
* deleted mbart50 by accident
* seamless m4t
* albert fix
* whisper
* layout3
* attempt to fix cached tokenizers on CI
* trying another fix on CI
* again try to work around CI
* bertweet
* tapas
* mbart50
* luke
* mluke
* markuplm
* markuplm
* fix some more auto tests
* some random model failures
* mistralcommontestser
* more fixes
* ref fix
* siglip
* marian
* plbart
* update utils toks
* seamless m4t
* roc bert
* udpate byt5 test
* xlm
* esm
* roformer
* code llama
* biogpt
* m2m100
* dpr and flaubert
* xlm and speech to text
* tok backend pass object
* tokenizer object pass
* wav2vec2
* wav2vec2
* cpmant
* update utils tokenizers
* cpmant
* bartpho
* test apply chat template assistant mask
* apply chat template video
* apply chat template assistant mask
* test torch
* update from slow in base and fix donut processor errors
* auto to point to tokenizers backend, fix kosmos2
* some non model fixes for old slow models that no longer have their own tokenizer file as they are the same as bert
* missed file from last commit
* idefics2
* fixup
* fixup
* pretrained tokenizer fast test update
* stash
* bad merged
* cherry pick more stuff that did not merge well
* fix gptsw3
* nit warn for now
* update error raising
* just ran fixup
* bring back bert legacy
* fix
* nit
* fix 56 errors on blenderbotsmall?
* 18 for blenderbotsmall
* tok auto
* missed clip
* fix tests
* something missed
* token healing
* tok common tests update - nonmodel
* try to fix non-model test in test_tokenization_utils
* fix hub tests
* try to fix hub tests
* custom vocab related fixed
* bert jap
* BERT JAP
* rename bert legacy to bert legacy
* Wav2vec2
* fix in tok python to update total vocab size - fixes speech t5
* blender bot small
* forgot test file
* test failures
* marian
* gpt2 tiktoken
* big bird / marian
* udop
* forgot couple changes
* test_serve fix
* missing import
* a couple processors fixes
* style partly
* fix to fetch tests ci
* Revert branch back to commit f5bc69ef state
* revert branch to styling
* update mistral after merge
* fixes for non model tests
* some processor test fixes
* more processor test fixes
* more processor fixes
* hub tests
* python tok utils
* fix hub test
* make style for now
* remove problemattic fic copies
* python utils/check_copies.py --fix_and_overwrite
* more styling
* fixup
* silence docstirng
* fix import?
* fix imports
* add the local test as well
* throw spm error
* llamas
* fix a couple tests
* broke ci
* broke ci
* broke ci
* broke ci
* add logs to debug gemma on ci
* gemma and llama
* gemma
* revert las commit
* gemma debug
* gemma debug
* gemma
* safely import spiece backend
* tok tests
* check none
* setup and qual
* ruff
* del dev files
* tok auto
* fill docstrings
* update auto
* blenderbot small nit
* add migration guide
* move mixtral patch to `TokenizersBackend`, move `TokenizerExtractor`
* rename MistralCommonTokenizer to MistralCommonB ackend
* nit
* fix failures
* fixup
* remoove one old test
* mark the slow one as slow
* very small fixes
* update auto mapping for missing ones
* fixup lorsd
* fixup doc and stuff
* should be the final fixe
* processing update
* update
* FIX or brute AI fix the llava test
* style
* slow?
* fix is offline mode?
* fix mt5
* One tok utils (#42462)
* consolidate python and utils tokenization files, they are copies
* ruff and ref
* Format
* fix cohere
* ?
* up
* am I dumbb?
* grumble
---------
Co-authored-by: Arthur <arthur.zucker@gmail.com>
* Add create token type ids to CodeGenTokenizer
* Fix inconsistent length of token type ids
* Format source codes
* Fix inconsistent order of methods
* Update docstring
* add test_tokenizer_integration test
* Format source codes
* Add `copied from` comment to CodeGenTokenizerFast
* Add doc of create_token_type_ids_from_sequences
* Make return_token_type_ids False by default
* Make test_tokenizer_integration as slow test
* Add return_token_type_ids to tokenizer init arg
* Add test for tokenizer's init return_token_type_ids
* Format source codes
* fix test for bart. Order is correct now let's skip BPEs
* ouf
* styling
* fix bert....
* slow refactoring
* current updates
* massive refactoring
* update
* NICE!
* update to see where I am at
* updates
* update
* update
* revert
* updates
* updates
* start supporting legacy_save
* styling
* big update
* revert some changes
* nits
* nniiiiiice
* small fixes
* kinda fix t5 with new behaviour
* major update
* fixup
* fix copies
* today's updates
* fix byt5
* upfate
* update
* update
* updates
* update vocab size test
* Barthez does not use not need the fairseq offset ids
* super calll must be after
* calll super
* move all super init
* move other super init
* fixup
* nits
* more fixes
* nits
* more fixes
* nits
* more fix
* remove useless files
* ouch all of them are affected
* and more!
* small imporvements
* no more sanitize token
* more changes around unique no split tokens
* partially fix more things
* keep legacy save but add warning
* so... more fixes
* updates
* guess deberta tokenizer could be nuked
* fixup
* fixup did some bad things
* nuke it if it breaks
* remove prints and pretrain fast from slow with new format.
* fixups
* Apply suggestions from code review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* fiou
* nit
* by default specials should not be normalized?
* update
* remove brakpoint
* updates
* a lot of updates
* fixup
* fixes revert some changes to match fast
* small nits
* that makes it cleaner
* fix camembert accordingly
* update
* some lest breaking changes
* update
* fixup
* fix byt5 and whisper mostly
* some more fixes, canine's byte vocab
* fix gpt2
* fix most of the perceiver tests (4 left)
* fix layout lmv3
* fixup
* fix copies for gpt2 style
* make sure to only warn once
* fix perciever and gpt2 tests
* some more backward compatibility: also read special tokens map because some ppl use it........////.....
* fixup
* add else when reading
* nits
* fresh updates
* fix copies
* will this make everything faster?
* fixes
* more fixes
* update
* more fixes
* fixup
* is the source of truth right?
* sorry camembert for the troubles
* current updates
* fixup
* update led
* update
* fix regression
* fix single word
* more model specific fixes
* fix t5 tests
* fixup
* more comments
* update
* fix nllb
* rstrip removed
* small fixes
* better handle additional_special_tokens and vocab sizes
* fixing
* styling
* fix 4 / 21
* fixup
* fix nlbb's tests
* some fixes
* fix t5
* fixes
* style
* fix canine tests
* damn this is nice
* nits
* m2m100 nit
* fixups
* fixes!
* fixup
* stash
* fix merge
* revert bad change
* fixup
* correct order for code Llama
* fix speecht5 post merge
* styling
* revert source of 11 fails
* small nits
* all changes in one go
* fnet hack
* fix 2 more tests
* update based on main branch of tokenizers
* fixup
* fix VITS issues
* more fixes
* fix mgp test
* fix camembert issues
* oups camembert still has 2 failing tests
* mluke fixes
* decode fixes
* small nits
* nits
* fix llama and vits
* fix camembert
* smal nits
* more fixes when initialising a fast from a slow and etc
* fix one of the last test
* fix CPM tokenizer test
* fixups
* fix pop2piano
* fixup
* ⚠️ Change tokenizers required version ⚠️
* ⚠️ Change tokenizers required version ⚠️
* "tokenizers>=0.14,<0.15", don't forget smaller than
* fix musicgen tests and pretraiendtokenizerfast
* fix owlvit and all
* update t5
* fix 800 red
* fix tests
* fix the fix of the fix of t5
* styling
* documentation nits
* cache _added_tokens_encoder
* fixups
* Nit
* fix red tests
* one last nit!
* make eveything a lot simpler
* Now it's over 😉
* few small nits
* Apply suggestions from code review
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* updates that work for now
* tests that should no be skipped / changed and fixed next
* fixup
* i am ashamed
* pushe the fix
* update
* fixups
* nits
* fix added_tokens_encoder
* fix canine test
* fix pegasus vocab
* fix transfoXL
* fixup
* whisper needs to be fixed for train new
* pegasus nits
* more pegasus fixes
* minor update
* better error message in failed test
* fix whisper failing test
* fix whisper failing test
* fix pegasus
* fixup
* fix **** pegasus
* reset things
* remove another file
* attempts to fix the strange custome encoder and offset
* nits here and there
* update
* fixup
* nit
* fix the whisper test
* nits nits
* Apply suggestions from code review
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* updates based on review
* some small update to potentially remove
* nits
* import rlu cache
* Update src/transformers/tokenization_utils_base.py
Co-authored-by: Lysandre Debut <hi@lysand.re>
* move warning to `from_pretrained`
* update tests results now that the special tokens are always added
---------
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: Lysandre Debut <hi@lysand.re>
* Result of black 23.1
* Update target to Python 3.7
* Switch flake8 to ruff
* Configure isort
* Configure isort
* Apply isort with line limit
* Put the right black version
* adapt black in check copies
* Fix copies
* Add CodeGen model
* Add missing key and switch order of super()
* Fix torch.ones init with uint8 instead of bool
* Address comments: copy statements and doc
* update tests
* remove old model parallel
* fix batch gen tests
* fix batch gen test
* update test_gpt2_sample_max_time
* fix codgen test and revert gpt2 test change
* Fix incorrect tie_word_embedding value, typo, URL
* Fix model order in README and styling
* Reorder model list alphabetically
* Set tie_word_embedding to False by default
* Apply suggestions from code review
* Better attn mask name & remove attn masked_bias
* add tokenizer for codegen
* quality
* doc tokenizer
* fix-copies
* add CodeGenTokenizer in converter
* make truncation optional
* add test for truncation
* add copyright
* fix-copies
* fix fast tokenizer decode
* Update src/transformers/models/codegen/tokenization_codegen.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* increase vocab_size in tests
Co-authored-by: patil-suraj <surajp815@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
