* server: expose the reasoning effort levels of the chat template
The capability probe renders a user turn for each candidate level
(minimal, low, medium, high, xhigh, max) and for an unknown level. A
level is honored when it renders without raising and differently from
the unknown level, which shows the template fallback. Levels sharing a
render are aliases and the highest one is kept.
The caps serialize to JSON, and chat_template_caps in /props reports
the list as reasoning_effort_levels, from lowest to highest.
* jinja: keep the caps free of the json dependency
The caps keep reporting their flags as a map, and the effort levels
stay a plain member. common_chat_templates_get_caps() builds the JSON
the server reports from both, so jinja depends on json only for the
probe inputs.
In --models mode the thread count per model is computed and printed,
but never set in the params, so every context runs on the common
default, the number of physical cores. With several models in
parallel this oversubscribes the CPU, and the busy-wait barriers of
the CPU backend make it collapse.
Set the count in cpuparams and cpuparams_batch.
* common : support deferred tools and tool_reference content parts
Assisted-by: Claude Opus 5.5
* common : credit the deferred tools co-author
Co-authored-by: Aldehir Rojas <765227+aldehir@users.noreply.github.com>
Assisted-by: Claude Opus 5.5
* tests : fix the type check of the deferred tools test
Assisted-by: Claude Opus 5.5
* model : add LiquidAI/LFM3-1.7B and LiquidAI/LFM3-VL-2B
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* tests : set the SWA rope base of the LFM3 test model
Assisted-by: Claude Opus 5.5
* llama : enable recurrent state rollback for LFM3
Assisted-by: Claude Opus 5.5
* Apply batched suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Tarek Dakhran <github@dakhran.com>
* gguf-py : align the lfm3vl mapping comments
Assisted-by: Claude Opus 5.5
* llama : fix CRLF line endings from the review suggestions
Assisted-by: Claude Opus 5.5
* tests : add the LFM3 Metal fusion baseline
Assisted-by: Claude Opus 5.5
* llama : keep the LFM3-VL depthwise conv kernels in F32 when quantizing
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* tests : speed up tensor data generation and model saving
Refactor save_models to create only the model, not a llama_context,
which is not needed for saving. Replace the slow std::normal_distribution
draw with a splitmix64 + box-muller normal generator.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : use a shared pool of normal draws for tensor data
Generate a pool of normal draws once with std::mt19937 and
std::normal_distribution, then have every tensor copy a cyclic slice of
it instead of running a per-tensor box-muller. The name-keyed slice
selection keeps the draw deterministic.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : log the size of saved models
Report the produced file size in bytes and MB after each save.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : simplify the model save log
Put the model name, target and size on one line per saved model.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* vulkan: allow dim 3 broadcast in MUL_MAT
supports_op rejected every MUL_MAT whose src0 and src1 differ in ne3,
although mul_mm, mul_mmq and mul_mat_vec already broadcast src0 over
dim 3 through broadcast3. Such a matmul then ran on the CPU with a
copy of the weight, and the split layout changed with the number of
streams in the ubatch.
Keep the strict check for MUL_MAT_ID only, and route the broadcast
cases away from mul_mat_vec_nc, the one path that requires equal ne3.
* tests : harden MUL_MAT with permuted dim 3 broadcast cases
Harden the MUL_MAT coverage with a weight broadcast over dim 3 against
a permuted per-stream activation, the shape of an MLA output
projection with a non unified KV cache, for a single token, several
heads and several tokens per stream.
* model : add MiniCPM-OCR-Jev support
The model is a Qwen3.5-VL trunk with a value head read at the last token: it
scores an OCR text against an image rendered from it, it does not generate.
Add the openbmb-minicpm-ocr-jev decision type, served by POST /v1/systemone.
The class scores are the pooled output of the sequence, read with last pooling,
so the conversion turns score.weight into cls.output and sets
embedding_length_out. The checkpoint has no LM head, so text generation is
rejected instead of using the output weights duplicated from the token
embeddings.
The task is fixed, so its prompt is entirely in the systemone template, the
same way h2o-lightning keeps its default there. The processor of the checkpoint
allows up to 16384 merged tokens per image, more than the Qwen-VL default of
mtmd, so the mmproj keeps the image budget of the preprocessor config. The
checkpoint declares MiniCPMOCRForSequenceClassification in its config.json, as
other models do.
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* fix code
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* use gguf meta classifier_output_labels
* remove redudant code changes
---------
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ggml : relax scale to non-contiguous views
SCALE requires src0 to be 1D-contiguous. Relax this to dim-0
contiguity with arbitrary row strides (ggml_is_contiguous_rows),
same as the unary ops in the Metal backend.
- ggml: relax the assert in ggml_scale_impl
- cpu: address src0 rows through their byte strides (f32, bf16)
- cuda: stride-aware kernel, fastdiv row decomposition
- metal: support bf16 scale (guarded unary instantiations)
- sycl: keep the gate at contiguous, the kernel is flat f32
Add view variants to test_scale. Verified on CPU, Metal and CUDA.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* cuda : skip the softcap fusion on non-contiguous views
The fused softcap kernel reads src and writes dst as dense arrays,
so a view reaching SCALE -> TANH -> SCALE now that SCALE accepts
non-contiguous rows would be read wrongly. Keep such graphs on the
strided kernels until the fusion becomes stride aware.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* tests : fix multi-seq split replay against trained models
The rollback snapshot is captured by the graph that decoded the rolled-back
tail, while the reference context decodes the same prefix with different
batch shapes. The two states differ in the last bits and trained models
amplify the difference with every replayed token. Compare only the first
replayed token of each seq, then restore the rollback state into the
reference context before the seq-1-only tail.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : require bit-exact seq-1 isolation in split replay
The seq state save is not always bit-lossless - SWA-masked cells are not
saved - so reload the restored state back into the rollback context. That
way both contexts enter the seq-1-only tail with byte-identical memory,
including the cell layout, and the tail logits must match bit-exact. Also
assert that the restored state round-trips bit-identical.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : drop the n_swa partial-state check from ps
The check keys off llama_model_n_swa(), which returns 0 for dsv4 even
though its raw KV cache has a 128-cell sliding window. The partial state
grows with the tokens there by design, so the flatness check is wrong.
Keep the partial <= full invariant.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Every dst row comes from src0 or src1, or for dim 0 from the row of
src0 followed by the row of src1, so a single kernel covers every
type: a dense source row is one memcpy, quantized types included,
and a strided source is copied element by element with the element
size known at compile time.
Rows are split across threads instead of i2 alone, which is often 1,
and are copied in tiles of 32: for a transposed source the elements
are walked in the outer loop, so the rows of a tile read neighboring
addresses instead of jumping a full row stride per element.
Add transposed b cases to test_concat, the layout of the delta-net
conv input, and a perf case for it.
* tests: pin model resolution on real repo listings
Extend test-model-resolution with 63 repo listings captured verbatim
from the HF API, chosen to cover every file naming pattern found across
1237 GGUF repos of the major publishers and pruned to whole quant groups.
Each repo pins its resolved plans and its cache listing.
The expectations follow what each file really is per its GGUF header,
not what the current resolver picks, so a mismatch is a resolution bug.
Mismatches are collected and reported together before the test fails.
* tests: leave the listing unchecked on paths the filesystem refuses
Some real split names (DevQuasar's Model.Q2_K.gguf-00001-of-00012.gguf
under a quant folder) exceed MAX_PATH once materialized in the temporary
cache on Windows, so the listing of such a repo is skipped with a note
while its plans stay checked.
* tests: pin real listings to master on GGUF convention names only
Replace the real repo table with 171 HF listings whose file names follow
the GGUF naming convention (lowercase sidecar prefix, - delimited
components, quant last), one per layout pattern found across 1174 GGUF
repos and pruned to whole quant groups. Repos outside the convention are
dropped. Each repo pins the plans and the cache listing master gives
today, so the test is a pure harness: any change in resolution shows up
here, and mismatches are reported together.
test-llama-archs included src/llama-arch.h and src/llama-model-saver.h,
which prevented building on Windows with shared libraries since the
used symbols are not exported from the DLL.
- declare llama_arch_* wrappers around the internal llm_ arch query
functions in the llama-ext.h staging header, together with
llama_arch_kv_name() and llama_arch_supports_save()
- slim llama_model_saver down to the private implementation of
llama_model_save_to_file(): the fixture-writer half used by the test
moves into tests/test-llama-archs.cpp as a local kv_writer
- replace the roundtrip saver usage with the public
llama_model_save_to_file() writing to a temp path (also removes the
tmpfile() fd leak and its Windows workaround)
- move test-llama-archs, test-generate-models, test-llama-context and
test-fusion out of the NOT WIN32 OR NOT BUILD_SHARED_LIBS guard
Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
* Metal : stop overflow from happening in large matrix output offsets
Assisted-by: Codex
* metal : widen batch offsets and MUL_MAT_ID token indices
Assisted-by: Codex
---------
Co-authored-by: feni6 <270037094+feni6@users.noreply.github.com>
* kv-cache : use ggml_backend_tensor_copy to refill the DSV4 K mirrors
* llama : move the Engram pad id to llama_hparams and the token map to llama_model
The saver no longer needs the deepseek41 model class.
* deepseek41 : drop the F32 src1 request on the HC mixes matmul
* ggml : add softplus TODOs in the backends that still use log(1 + exp(x))
* convert : make the DSV4/DSV4.1 modify_tensors generators, drop the redundant vision filter
* deepseek41 : clean up comments
* deepseek41 : drop the F32 src1 requests on the router and indexer matmuls
* convert : use format_tensor_name directly in DeepseekV4Model
* s/comp_kv_src/comp_il_src/g
* cont : fix TODOs for log1p
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* common, grammar: add recursive depth limit, make it a requirement
* catch more recursive functions
* common : tidy up peg parser changes
* grammar : reject candidates in a loop, without a depth limit
The depth limit in llama_grammar_reject_candidates_for_stack counted
the code points of a vocab token, so any token of 256 chars or more
was always rejected under a grammar, such as the 512-space token of
GLM-4.7-Flash or the long space tokens of Laguna-S-2.1.
The rejection now walks the code points with its own stack of levels
instead of recursing: there is no limit and no stack risk, whatever
the length of a token from the GGUF, and the masks match master.
* grammar : track rejected candidates as indices into the input
At a given code point a candidate only differs from its input by the
position of its code point pointer, so the levels now carry indices
into the input candidates instead of copies. The first level needs no
special case and the rejects map back to the input directly.
Grammar sampling gets faster than master on GLM-4.7, Laguna, Qwen3.6
and gemma-4 with the json, list and spaces grammars, same masks.
* nit : shorten the comment of llama_grammar_reject_step
* common : add depth_guard for schema walks
* grammar : move GRAMMAR_RECURSIVE_LIMIT into the .cpp
* common : reduce recursion limit for peg parser
* improve
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
Co-authored-by: Pascal <admin@serveurperso.com>
* llama-quant : fix quantizing 4D tensors
nrows_total was ne[1]*ne[2], so for a 4D tensor only the first ne[2]
slices were quantized and the rest of the rows were dropped. Use
ggml_nrows(tensor) to count all rows, including ne[3].
Fixes#29916
* llama-quant : detect expert tensors by name and fix imatrix size for 4D tensors
The MXFP4_MOE path used ne[2] > 1 to detect MoE expert tensors. This is
wrong for non-expert tensors with 3 or more dims, e.g. MLA projections
(BailingMoE3) and 4D conv kernels. Use the tensor name instead: all
tensors used by GGML_OP_MUL_MAT_ID are named *_exps or *_chexps. This
also removes the BailingMoE3 special case.
The imatrix holds one vector of ne[0] values per 2D slice, so the
expected size is ne[0]*ne[2]*ne[3], not ne[0]*ne[2]. This matches how
the imatrix is indexed per row for 4D tensors in llama_tensor_quantize_impl.
Assisted-by: Claude Opus 5.5
* cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows
For prompt processing, run the gate and up matmuls of a dense FFN as one
mul_mat_q launch and apply the GLU in the write-back.
- The loaders take the gate weights as a second pointer and fill the weight
tile with up and gate rows interleaved in chunks of 8, so each thread holds
an up value and the matching gate value (rows r and r + 8 of an MMA tile).
- The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[]
per thread. The fused write-back computes the GLU from registers and writes
half the rows.
- The fused path uses the regular tiling (no stream-k) with the column tiles
innermost so that neighbouring blocks share weight tiles.
- Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer.
* cuda: assert the fused MMQ gate argument and trim redundant comments
* cuda: move the fused MMQ hint check to should_fuse and add TODOs
* cuda: pass prec_src1 to the fused MMQ write-back
* cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back
* cuda: require has_fusion explicitly, drop the fusion-only src1 precision check
* cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile
* cuda: inline the MMQ tiling-efficiency check
* tests: one fused gate/up MMQ case per type and GLU op
* cuda: always launch the fused MMQ kernel with one block per tile
* tests: shorten the fused gate/up MMQ test comment
* cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs
* cuda: run the fused MMQ kernel on the plain tiling path
* cuda: put the column tiles innermost in the MMQ plain tiling path
* vulkan: Add a larger tile size for conv3d (coopmat2 only)
This adds a larger tile size, in practice it is only for coopmat2 where
we don't need shared memory to store the output (would use more than 64KB).
Add some additional representative backend tests. At least, this is what codex
came up with from looking at sd.cpp Wan VAE.
* only apply large tile size for newer driver versions
* chat : reject JSON nested more than 128 levels before walking it
The schema builder, the jinja engine and the decision code walk request
JSON with recursion. A request nested deep enough overflows the stack of
the HTTP thread and llama-server dies (SIGBUS on macOS, SIGSEGV on Linux):
for a tool schema from a few hundred levels on macOS and about 10000 on
Linux.
Check the depth first and answer 400 past 128 levels: the chat inputs in
common_chat_templates_apply (tools, tool-call arguments,
chat_template_kwargs, json_schema), the json_schema of /completion, and the
whole /v1/systemone body. A gemma4 tool result that is too deep stays text.
* fix the root, not the call side
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add DeepSeek-V4.1-Flash (deepseek41)
* deepseek41 : remove MXFP8, convert the Engram table to BF16
* deepseek41 : compute the compressed-KV read layers once in the cache
* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le
* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather
* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer
* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle
* deepseek41 : keep il last in build_hc_mixes
* deepseek41 : sort the top-k rows before the sparse gather
* tests : add deepseek41 to the Metal fusion baseline
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* hexagon: KV-head team split for HMX flash attention in multicore
* hexagon: merge FA_HEAD_SPLIT and FA_SPLIT into single GGML_HEXAGON_FA_SPLIT flag
0 = off (token-split only); 1 = head-split, plus team-split when
n_kv_heads < N_cores so stranded cores still participate (e.g. gemma-4
with 2 KV heads on 4 cores). Where heads divide evenly across cores,
plain head-split is used.
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the
per expert-token amax scaling done in the compute graph
(ref: https://github.com/ggml-org/llama.cpp/pull/30861), which keeps the
MoE down-projection activations within f16 range before the op runs.
Drop the two-stage reduction kernels, their scratch buffer and the
GGML_PREC_F32 dispatch added in https://github.com/ggml-org/llama.cpp/pull/26223,
as well as the test cases that drive MUL_MAT_ID src1 out of f16 range.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* ggml : allow MUL_MAT_ID -1 index to skip the computation
* nits
* update vulkan code
* ggml-webgpu: ensure all threads reach workgroup barriers in mul_mat_id_vec
(cherry picked from commit d8a4b9efb6e2b9abf3a866e847756b7e6bbc6b69)
* fix cuda
* openvino
* exclude CUDA from this PR
* add TODO tag
---------
Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
* tests : add speculative rejection draft sampling test
This commit adds a test for speculative rejection sampling test that
excercises the speculative rejection sampling mechanism in
common/speculative.cpp.
The motivation for this is have a test that verifies that the
speculative draft models emitts tokens that match the target models
distribution. So when rejection sampling occurs it should still
produce the same probability distribution as if we directly sampled
from the target model.
Running the test:
```console
$ ctest --test-dir build -R "^test-speculative-rejection-sampling$" \
--n_trials 5000 -V
```
This will generate two models, one target and one draft model.
For subsequent runs use the "fixture exclude any" option to avoid
recreating the models:
```console
$ ctest --test-dir build-cpu-debug -R \
"^test-speculative-rejection-sampling$" \
-FA "^speculative-models$" --n_trails 5000 -V
```
* remove draft model
* mtmd : lfm2 bilinear resize matches torchvision
Lfm2VlImageProcessor and d1-omni vision.py resize uint8 images with
torchvision (antialias=True). That is Pillow's algorithm, except that the
weights are int16 with as many fractional bits as the largest weight allows,
where Pillow uses 22 bits. The Pillow port was 1-2 levels off on about 1% of
the values of the d1-omni and LFM2.5-VL crops.
RESIZE_ALGO_BILINEAR_TORCH rounds the weights of the Pillow port to that
precision and scales them back to 22 bits, which gives the output of
torchvision with the same passes, and "bilinear" maps to it for LFM2 and
d1-omni. Other models keep the Pillow algorithms.
* mtmd : lfm2 bicubic resize matches torchvision
Lfm2VlImageProcessor resizes with torchvision for the bicubic checkpoints too
(d1-3B, LFM2-VL), so "bicubic" maps to RESIZE_ALGO_BICUBIC_TORCH for LFM2: the
Pillow bicubic filter (a = -0.5, as torchvision with antialias) with the int16
weights of torchvision. The Pillow port was 1-2 levels off on about 0.25% of
the values of the d1-3B crops.
* use bitfield, add hparams setter
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* qwen4exp : build the QSA selection without the kq_mask
* clean
* lang
* qwen4exp : reshape the fused QSA selection to the kq_mask streams
ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while
build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stream].
With several streams the shape assert failed at graph reserve.
* test the QSA optimization
* reword
* update build_fused_top_k_mask instead of handwiring
* simplify
* keep il at the end for consistency
* hex-fusion: support for GELU+MUL, REPEAT+GET_ROWS and BINARY_NX (back-to-back ADD/MUL/...)
* hex-fusion: factor out fusion logic into separate .h and .cpp and update all related code
* hex-fusion: allow for RMS_NORM+MUL fusions for non-contig tensors
* hex-fusion: streamline fusion eligibility checks and cover missing corner cases
* hex-fusion: streamline fusion flow
* hex-opbatch: make n_tensors limit more scalable
* hex-bufs: adjust the trace buff sizing
* hex-glu: cleanup glu-ops interfaces and add kernel_params
* hex-kparams: clean up kparams handling in unary, ssm-conv, and other ops
* hex-kparams: update the rest of the ops PAD,DIAG,... to use kparams
* hex-unary: add support for collapsing dims
* hex-ops: cleanup redundant runtime checks
* hex-ops: consistent use of kernel_type in all kparams
* hex-fusion: remove redundant check in gdn+cpy
* hex-fusion: further streamlining of fusion rules and code
* hex-fusion: cleanup naming
* hex-opfmt: cleanup op formatting
* hex-fusion: bump fusion lookahead to 64
* hex-fusion: fuse SSM_CONV+SILU
* hex-fusion: fix hvx register spills in fused ssm_conv+silu
* hex-fusion: add more GLU fusions
* hex-fusion: remove duplicate checks and use ggml-common helpers
* hex-kparams: refactor kparams into htp-kparams.h/cpp
* hex-fusion: fix binary-nx fusion regression
* hex-kparams: another pass at cleaning up kparams (removing unused, etc)
* hex-kparams: streamling nameing and move _check_ wrappers into .h
* hex-kparams: templetize htp_kparams_check_<op> to reduce code bloat
* hex-ops: de-duplicate op dispatch switches
* hex-kparams: trim down exposed kparams calls
* hex-kparams: simplify type casting of kparams structs
* hex-fence: de-dup fence tensor construction
* hex-opbatch: simplify tensor shape init/checks
* hex-mm: cleanup leftovers of the original QK defines
* hex-fa: fix stack overflow on older npu firmware
* hex-ctx: move dma_caches into ops_ctx
* Multi-column on Intel XMX matrix engines refactored
Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0).
* Fixed regression bug
real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4
* Removed GGML_TYPE_Q8_0 activation
* rewritten xmx_mul_mat
64-byte alignment and 16-byte pitch check on the activations
* Implemented fused gate/up kernel
One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types.
* Removed the temporary XMX_NOGLU environment switch
* Q2_K min-sum optimization
Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group
* Added dispatch check and GRF sweep
tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%).
Q6_K is unchanged.
MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361.
Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without.
* Renamed variables
* Added 2 environment variables
GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it.
* Update esimd.hpp
1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces.
2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once.
The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
* Add Flint model (FlintForCausalLM). Assisted by Codex
* cleanup
* one more
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : add SenseNova U1 understanding support
Add support for the dense SenseNova U1 understanding branch, including
the U1.5-8B-MoT model.
- Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation
tensors and concatenating temporal and spatial Q/K normalization
weights.
- Register the sensenova_u1 GGUF architecture and custom spatial RoPE
metadata.
- Implement temporal/spatial Q/K normalization with 4-axis RoPE.
- Keep U1-specific text position handling in the model implementation.
- Add synthetic architecture coverage for sensenova_u1.
Only the dense understanding branch is supported; generation
branches are intentionally rejected during conversion.
* conversion : treat one-expert SenseNova U1 configs as dense
* mtmd : add SenseNova U1 vision support
Add conversion support for the SenseNova U1 vision encoder and implement
its CPU mtmd projector graph.
The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial
downsampling, and dense projection for dynamic-resolution image inputs.
* mtmd : fix SenseNova U1 vision token layout
* model : clean up SenseNova U1 RoPE metadata
Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV
definitions used by conversion, model loading, and architecture tests.
Clarify why SenseNova U1 retains its custom text-position input: U1 uses
[t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0].
* mtmd : simplify SenseNova U1 projector integration
Map the SenseNova U1 dense projection to the standard mm.0 weight and
bias tensors and reuse the shared projector fields in the loader and graph.
Read vision parameters from the checkpoint configuration and remove the
U1-specific tensor names and model fields while preserving its strided
projection, bias, and row-major token layout.
* mtmd : fix SenseNova U1 image positions and attention
* fix spacing
* mtmd : restore generic test input chunks
* mtmd : restore SenseNova U1 integration after upstream merge
* Update conversion/sensenova_u1.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : remove SenseNova U1 BF16 activation casts
Use standard backend precision handling for QKV, FFN, and output
projections instead of casting inputs based on the output tensor type.
Assisted-by: Codex
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama: create new fused op for QSA scoring
* enroll glm5.3-next too
* tests : initialize the remaining tensors in test_kpool_mask
The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN check.
* ggml: generalize the k-pool mask op into ggml_top_k_mask
The k-pool mask op only covered the pooled QSA selection of qwen4exp
and glm5-next, while the DSA models and the DeepSeek V4 CSA build the
same top-k mask by hand with fill, set_rows and add.
ggml_top_k_mask(mask, top_k, score, blocks, extra) keeps the mask
cells of the live picks and the extra cells, and sets the rest to
-inf. Every row tensor is 4D in the layout of top_k, and the mask is
broadcast to those rows, so a selection can also be made per head.
score, blocks and extra are optional: without score every pick is
live, without blocks a pick is a cell, and extra holds the cells a
row keeps without a pick.
llm_graph_context::build_fused_top_k_mask views the row tensors in
the layout of kq_mask and registers the fused node. The k-pool
models, the DSA attention of build_attn and the DeepSeek V4 CSA mask
use it when the probe enables the op, and keep their current path on
backends without it.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table
* ggml-cpu : keep the GGML_GELU_QUICK_FP16 table path
* ggml-cpu : add the VXE gelu_quick path
* tests : add test-gelu-quick for the f32 CPU kernels
* tests : drop inf/nan inputs from test-gelu-quick
* mtmd : lfm2 tiles stretch the image to the grid
Lfm2VlImageProcessor.crop_image_to_patches resizes the image straight to
(512 * rows, 512 * cols) before it cuts the tiles. llama.cpp kept the aspect
ratio and added black bars for LFM2 (only d1-omni stretched), so the tiles of
d1-3B and every LFM2-VL / LFM2.5-VL model differed from transformers unless
the image had the aspect ratio of the grid. With PAD_NONE the tiles are the
same as in transformers.
* mtmd : lfm2 layout with Python rounding in double precision
Lfm2VlImageProcessor.smart_resize rounds with Python's round (half to even)
and computes beta and the tile grid aspect ratio in double. llama.cpp used
std::round and float, so 31926 of the 16.7M sizes in [1,4096]^2 got another
thumbnail size or grid than transformers (and d1-omni vision.py), e.g.
400x600 -> 416x608 instead of 384x608 and 720x1600 -> 2x5 tiles instead of
2x4. The lfm2 smart resize and grid choice now follow the Python code and
match it on every size of that range.
* fix
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* mtmd : read LFM2 tiling params from GGUF metadata
Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from
processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp
with the previous constants as defaults. do_image_splitting=false maps to
min_tiles=max_tiles=1 (no tiling).
Rebased version of #25524, original implementation by Michael DM (@michael-dm).
Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
* fix test
---------
Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
* metal : optimize argmax and pool_1d
Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : add pool_1d perf cases
Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : minor fix
Pascal