MattandArthur d0deac8f87 Add chat input sanitization (#47386) ...

* Cleanup and huge simplification to an alternate approach

* Cover all control tokens, not just special tokens

* Comment cleanup

* Comment cleanup

* Refactor everything out into chat_template_utils, couple of cleanups

* Add FIXME

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

2026-10-13 13:49:51 +00:00

..

__init__.py

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

test_tokenization_fast.py

QA: applied ruff rule PLW1514 (#48990)

2026-09-22 06:49:19 +00:00

test_tokenization_utils.py

Add chat input sanitization (#47386)

2026-10-13 13:49:51 +00:00