Kashif RasulandSigbjørn Skjæret 4c3c5810f8 hybrid dna tokenizer fixes (#30825) ...

* vocab : support Carbon-14B HybridDNA tokenizer

Carbon-14B auto-tags untagged DNA runs and uses N-padded tail tokens
(tail=pad_n). Also strip whitespace inside <dna> tags and handle a stray
</dna> like the HF tokenizer.

* convert : fix special tokens and merges for older Carbon models

Carbon-500M/3B/8B don't register the Qwen added tokens and ship no
tokenizer.json, so take the token types, merges and special ids from the
base tokenizer.

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* address review comments

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

2026-10-19 23:01:40 +02:00

..

__init__.py

model : add Kolibri-1 (Aleph-Alpha) support (#30673)

2026-10-17 13:00:17 +02:00

afmoe.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

arctic.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

baichuan.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

bailingmoe3.py

model : add Ling 3.0 VL support (#29151)

2026-09-24 08:57:31 +02:00

bailingmoe.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

base.py

hybrid dna tokenizer fixes (#30825)

2026-10-19 23:01:40 +02:00

beam.py

model: add Beam (Reflection) (#30670)

2026-10-17 00:07:55 +02:00

bert.py

llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) (#29818)

2026-10-02 11:56:04 +02:00

bitnet.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

bloom.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

chameleon.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

chatglm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

clef.py

server: support vision input for Clef (#29969)

2026-10-05 14:02:40 +02:00

codeshell.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

cogvlm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

command_r.py

mtmd: add cohere2 vision support (#30062)

2026-10-07 20:07:50 +02:00

dbrx.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

deci.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

deepseek.py

convert : respect Mistral format for DeepseekV2Model (#30669)

2026-10-16 15:17:30 +02:00

dots1.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

dots3.py

mtmd: support dots3-note vision+audio (#27524)

2026-08-22 10:35:50 +02:00

dotsocr.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

dream.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

ernie.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

exaone.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

falcon_h1.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

falcon.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

gemma.py

model : fix DFlash output head sharing (#30111)

2026-10-08 17:42:28 +02:00

glm.py

add GLM-5.3-Flash (GLM5-Next) support (#27773)

2026-09-30 14:20:32 +08:00

gpt2.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

gpt_oss.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

gptneox.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

granite.py

model : Granite switch decision 256 options (#30693)

2026-10-17 09:14:32 +02:00

grok.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

grovemoe.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

hrm_text.py

model : add support for HrmTextForCausalLM (DFM Mimir 1B) (#27625)

2026-09-16 15:18:45 +02:00

hunyuan.py

spec : support DFlash for HunyuanOCR (#28890)

2026-09-22 15:04:52 +02:00

hy_v4.py

convert : refactor Hy4-preview conversion - move HC tensor mapping to the global map (#28451)

2026-09-07 15:20:58 +02:00

internlm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

internvl.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

jais.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

jamba.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

januspro.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

k2_horizon.py

model : add K2 Horizon dense and MoVA support (#29535)

2026-10-06 18:42:24 +02:00

kimi_k3.py

convert: add MiMo-V2.6 support (#29257)

2026-09-22 14:38:09 +02:00

kimi_linear.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

kimivl.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

kolibri1.py

model : add Kolibri-1 (Aleph-Alpha) support (#30673)

2026-10-17 13:00:17 +02:00

laguna.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

lev.py

model: support nimble decision model (#29844)

2026-10-02 14:56:53 +02:00

lfm2.py

model : add LiquidAI/d1-omni-600M decision model (#30114)

2026-10-08 02:07:49 +02:00

lighton_ocr.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

llada.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

llama4.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

llama.py

hybrid dna tokenizer fixes (#30825)

2026-10-19 23:01:40 +02:00

llava.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

maincoder.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

mamba.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

maple.py

llama: add Maple 20B-A1B ternary MoE architecture (CPU) (#27000)

2026-09-14 14:04:05 +03:00

mellum.py

model : MTP support for Mellum2.1 (#30612)

2026-10-16 10:20:03 +02:00

mimo.py

llama : fix tensor split for fused qkv with uneven K/V head sizes (#29294)

2026-09-25 11:15:33 +08:00

minicpm.py

model : support MiniCPM-V 4.7 (#29416)

2026-10-10 20:41:59 +02:00

minimax.py

ci : bump ty to 0.0.78 (#28548)

2026-09-07 21:10:06 +02:00

mistral3.py

mtmd: add mtmd_audio_stream + support voxtral-realtime (#30629)

2026-10-16 01:46:42 +02:00

mistral.py

convert : fix conversion for Mistral-Medium-3.5-128B (#24268)

2026-06-07 21:41:39 +02:00

mpt.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

muse_glimmer.py

model: add Muse Spark Open (text, vision and audio) (#30616)

2026-10-15 23:27:12 +02:00

muse_spark.py

model: add Muse Spark Open (text, vision and audio) (#30616)

2026-10-15 23:27:12 +02:00

nanbeige.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

nemotron.py

models : add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support (#30520)

2026-10-15 16:34:56 +03:00

olmo.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

openelm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

orion.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

pangu.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

phi.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

pixtral.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

plamo.py

convert : fix token configuration for PLaMo-3 (#29843)

2026-10-07 11:34:16 +02:00

plm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

pockettts.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

pplx_decider.py

convert : support text-only checkpoints in pplx-decider (#30575)

2026-10-16 08:56:00 +02:00

qwen3tts.py

qwen3tts : guard speaker_encoder_config patch for CustomVoice variant (#29179)

2026-10-07 12:57:48 +02:00

qwen3vl.py

llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) (#29818)

2026-10-02 11:56:04 +02:00

qwen4exp.py

Qwen4Exp: add MTP (#29761)

2026-10-01 14:13:27 +03:00

qwen.py

hybrid dna tokenizer fixes (#30825)

2026-10-19 23:01:40 +02:00

qwenvl.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

refact.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

rwkv.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

sarashina2.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

smallthinker.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

smolvlm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

spark2_5.py

[Model] Support for Spark2_5ForCausalLM implementation (#27868)

2026-09-06 17:43:58 +02:00

stablelm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

starcoder.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

step3.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

t5.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

talkie.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

ultravox.py

mtmd: add mtmd_audio_stream + support voxtral-realtime (#30629)

2026-10-16 01:46:42 +02:00

vllm_d3.py

model: support vllm d3 models (#30383)

2026-10-12 15:01:52 +02:00

vllm_decision2.py

model: support vllm decision 2 model (#30378)

2026-10-12 13:39:55 +02:00

wavtokenizer.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

xverse.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

youtuvl.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00