* feat(ui): add Show tool call in progress display setting
Add the showToolCallInProgress key and its checkbox to Settings > Display,
right next to its "always show" counterpart. Upstream dropped a setting of
this exact name in 43b5e63589, folding it into alwaysShowToolCallContent
because the old name promised expansion that outlives the call; this brings
back the narrower, in-progress-only meaning.
The entry is registry-driven, so SETTING_CONFIG_DEFAULT, the dialog field and
the ui_settings admin baseline pick it up without extra wiring. It ships
enabled, mirroring Show thought in progress, and the always-show help text is
reworded so the two toggles read as a pair rather than contradicting each
other.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)
* feat(ui): expand tool call blocks while they are in progress
Drive the default expansion of TOOL_CALL_STREAMING and TOOL_CALL_PENDING from
showToolCallInProgress and leave TOOL_CALL, the terminal state, to
alwaysShowToolCallContent. A finished call now folds back on its own once its
result lands, which is exactly how the reasoning block behaves when the
thought completes.
Manual toggling keeps precedence over both defaults in either direction:
isExpanded() only consults the settings while expandedStates holds no entry
for the section.
Client tests pin the truth table for the two toggles, including the automatic
fold-back and both manual overrides.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)
* fix(ui): keep running tool calls expanded
A call the agentic loop reports as running now outranks its section type.
exec_shell_command persists an empty result before reading any output, so a
live call is already TOOL_CALL and folded back, which hid the streaming
output the setting is meant to show.
Move the expansion default to isSectionExpandedByDefault so the running-id
rule is covered by a truth table, and test the pending -> finished transition
instead of only the finished state: a block that only ever got mounted
finished cannot prove it folds back on its own.
Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:Q8_0)
* feat(ui): stream the exec command from the first character
The exec parser read its args strictly, so while the model was still
typing the call the block title stayed empty and the body claimed
"Running...". Parse the args partially (the read_file / edit_file /
write_file pattern), keep a terminal cursor while the args JSON is
incomplete, and stand the conversation cwd in for the x-tool-cwd that
only arrives with the result message, so the prompt is there from the
first character. The body stays empty while the args stream, shows
"Executing command..." once the call is executing, and the wrapper
drops its padding for empty phases so the title sits flush with the
frame.
Assisted by: Delta (zai-org/GLM-5.3)
* fix(ui): stream file tool bodies even before a path arrives
write_file and edit_file refused to render until their args carried a
path, so a call that streams its body first (or a same-named tool with
other arg names) showed an empty block. Meta now carries optional
fileName/filePath, path alias reading moves to shared helpers, and the
blocks fall back to the raw args JSON when no known content key is
present.
Assisted by: Delta (zai-org/GLM-5.3)
* fix(ui): keep an empty write_file body instead of the raw args
A call carrying {"content":""} fell through to the raw-args fallback, so
a finished write of an empty file showed the args JSON. The parser now
leaves content undefined when the key is missing and the block tests
!== undefined, so an empty write renders as an empty code block with its
byte count.
Assisted by: DeepSeek-V4.1-Flash (deepseek-ai)
Two quants of one repo pulled each other's draft: the badge read every
--model-draft arg of the repo, so each row showed both drafts.
Assisted-by: DeepSeek-V4.1-Flash-1017
* opencl: add F32-to-I32 ARGMAX
Assisted-by: OpenCode
* opencl: init to 0 to be consistent with cpu
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* Add Flint model (FlintForCausalLM). Assisted by Codex
* cleanup
* one more
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : add SenseNova U1 understanding support
Add support for the dense SenseNova U1 understanding branch, including
the U1.5-8B-MoT model.
- Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation
tensors and concatenating temporal and spatial Q/K normalization
weights.
- Register the sensenova_u1 GGUF architecture and custom spatial RoPE
metadata.
- Implement temporal/spatial Q/K normalization with 4-axis RoPE.
- Keep U1-specific text position handling in the model implementation.
- Add synthetic architecture coverage for sensenova_u1.
Only the dense understanding branch is supported; generation
branches are intentionally rejected during conversion.
* conversion : treat one-expert SenseNova U1 configs as dense
* mtmd : add SenseNova U1 vision support
Add conversion support for the SenseNova U1 vision encoder and implement
its CPU mtmd projector graph.
The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial
downsampling, and dense projection for dynamic-resolution image inputs.
* mtmd : fix SenseNova U1 vision token layout
* model : clean up SenseNova U1 RoPE metadata
Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV
definitions used by conversion, model loading, and architecture tests.
Clarify why SenseNova U1 retains its custom text-position input: U1 uses
[t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0].
* mtmd : simplify SenseNova U1 projector integration
Map the SenseNova U1 dense projection to the standard mm.0 weight and
bias tensors and reuse the shared projector fields in the loader and graph.
Read vision parameters from the checkpoint configuration and remove the
U1-specific tensor names and model fields while preserving its strided
projection, bias, and row-major token layout.
* mtmd : fix SenseNova U1 image positions and attention
* fix spacing
* mtmd : restore generic test input chunks
* mtmd : restore SenseNova U1 integration after upstream merge
* Update conversion/sensenova_u1.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : remove SenseNova U1 BF16 activation casts
Use standard backend precision handling for QKV, FFN, and output
projections instead of casting inputs based on the output tensor type.
Assisted-by: Codex
* Update src/models/sensenova-u1.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama: create new fused op for QSA scoring
* enroll glm5.3-next too
* tests : initialize the remaining tensors in test_kpool_mask
The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN check.
* ggml: generalize the k-pool mask op into ggml_top_k_mask
The k-pool mask op only covered the pooled QSA selection of qwen4exp
and glm5-next, while the DSA models and the DeepSeek V4 CSA build the
same top-k mask by hand with fill, set_rows and add.
ggml_top_k_mask(mask, top_k, score, blocks, extra) keeps the mask
cells of the live picks and the extra cells, and sets the rest to
-inf. Every row tensor is 4D in the layout of top_k, and the mask is
broadcast to those rows, so a selection can also be made per head.
score, blocks and extra are optional: without score every pick is
live, without blocks a pick is a cell, and extra holds the cells a
row keeps without a pick.
llm_graph_context::build_fused_top_k_mask views the row tensors in
the layout of kq_mask and registers the fused node. The k-pool
models, the DSA attention of build_attn and the DeepSeek V4 CSA mask
use it when the probe enables the op, and keep their current path on
backends without it.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ggml-opencl : add IM2COL_3D support
Assisted-by: OpenCode
* opencl: use same type for kernel and host
* opencl: fix for older adreno
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* server : forward llama.cpp sampling params on /v1/messages
* server : pass the request through on /v1/messages instead of copying listed keys
* NIT: Change comment from multiline to single line
* server : accept llama.cpp params under extra_params on /v1/messages
* server : remove test case for trivial functionality
* server : show extra_params in the /v1/messages README example
* ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table
* ggml-cpu : keep the GGML_GELU_QUICK_FP16 table path
* ggml-cpu : add the VXE gelu_quick path
* tests : add test-gelu-quick for the f32 CPU kernels
* tests : drop inf/nan inputs from test-gelu-quick
* llama : scale moe down projection per expert token
Replace the global amax rescaling for the MoE down projection with a per-expert-token factor computed by a POOL_1D max reduction along the intermediate dimension. Undo the scale on the output with the same exact power-of-two factor. Handle POOL_1D in the Meta split-state logic so a split along axis 0 becomes a replicated per-device partial result, enabling tensor parallelism.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : simplify comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : refactor amax src1 scaling result
Return the rescaled activation and the optional scale tensor through a small struct instead of an output pointer.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : add TODO + clean-up
* mtmd: add audio out stream api
* add /tts endpoint
* add docs
* wire up mtmd_helper_model_can_chat
* upload speaker_ref via form-data
* expose seed to api
* fix header for pcm
* wip
* fix penalty_last_n
* fix server_output_limits
* fix pocket-tts stream
* wip
* add server_schema for tts
* add server routing
* server: coerce seed and stream from multipart form fields
Form fields arrive as strings, so -F seed=42 and -F stream=true were
rejected by the schema. seed joins the integer fields, parsed as a long
long so a full uint32 seed fits, and stream is read as a boolean when
it comes as a string.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server: add realtime transcription API
* drop transcription_sessions
* proxy ws
* add docs
* server: let the API key middleware guard the websocket
httplib runs the pre-routing handler before the websocket upgrade, so
the API key middleware answers 401 at handshake like any other route.
This covers the router proxy, whose children run keyless, and accepts
the same key forms as HTTP (Bearer, raw key, X-Api-Key).
* server: keep realtime slots out of the prompt loop
A realtime slot waiting for audio is GENERATING with nothing to decode,
so it fell through the prompt loop and took slot_batched. A request on
a higher slot then failed can_batch_with on the task type and starved
for the whole session. Realtime slots have no prompt: the loop sets
them up once and always leaves them.
* server: start the realtime session once its slot is live
The audio stream was created in the prompt loop, which skips a slot
that cannot batch with the one generating, and session.created went out
before the task was posted: early appends failed and the reader quit,
leaving the socket open and the slot held until the HTTP timeout.
The slot now sets up its stream at launch and signals it with the begin
result, session.created waits for it, and a reader that stops closes
the socket so the session ends with it.
* server: name the committed item in input_audio_buffer.committed
Per the Realtime spec, item_id is the item made from the committed audio
and previous_item_id the one committed before it, null for the first.
committed now comes before the transcript that completes that item, and
both go out with the item switch under one lock, so a delta cannot land
between them.
* server: use the GA transcription session shape
The session object follows the GA Realtime schema: type transcription,
with the input format, transcription model, turn detection and noise
reduction under audio.input. The rate stays the model sample rate, which
is what the client must send for now.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* llama : move LFM2 shortconv and FFN blocks into a shared LFM base
* Update src/models/lfm-base.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* return logprobs in spec path
* server: add regression test for logprobs with speculative decoding
Asserts accepted draft tokens report the target model's logprobs,
comparing a draft-enabled run against a no-draft baseline (#24271).
Fails on master (top_logprobs empty, logprob 0.0), passes with the
fix from #27196.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KARBWpPcijcBXzJEvttuwk
---------
Co-authored-by: Jeonghyun Lee <sig.jhl@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* mtmd : lfm2 tiles stretch the image to the grid
Lfm2VlImageProcessor.crop_image_to_patches resizes the image straight to
(512 * rows, 512 * cols) before it cuts the tiles. llama.cpp kept the aspect
ratio and added black bars for LFM2 (only d1-omni stretched), so the tiles of
d1-3B and every LFM2-VL / LFM2.5-VL model differed from transformers unless
the image had the aspect ratio of the grid. With PAD_NONE the tiles are the
same as in transformers.
* mtmd : lfm2 layout with Python rounding in double precision
Lfm2VlImageProcessor.smart_resize rounds with Python's round (half to even)
and computes beta and the tile grid aspect ratio in double. llama.cpp used
std::round and float, so 31926 of the 16.7M sizes in [1,4096]^2 got another
thumbnail size or grid than transformers (and d1-omni vision.py), e.g.
400x600 -> 416x608 instead of 384x608 and 720x1600 -> 2x5 tiles instead of
2x4. The lfm2 smart resize and grid choice now follow the Python code and
match it on every size of that range.
* fix
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model : add H2O-Lightning-4B decision model
Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b
(Qwen3.5-4B with a merged decision LoRA).
None of the existing types match this model. It answers with " A"
rather than "A", uses up to 255 labels, and was trained on Python
json.dumps text, so floats like 12.0 must render as 12.0.
The converter writes the decision metadata and the systemone template
from the model's serve_config.json and refuses a config that doesn't
match. Tests skip unless the tiny test model is available locally.
Assisted-by: Claude Opus 5.5
* rm redundant code
* put labels to gguf
---------
Co-authored-by: Jonathan C. McKinney <jon.mckinney@h2o.ai>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* Router wasn't propagating the client disconnects fully to the child, the response wait could cause a delayed cancellation
* tests: satisfy router disconnect type check
* server : remove timing-sensitive router cancellation test
Assisted-by: Codex
* server: include output_index in Responses API SSE events
* server: reuse Anthropic streaming code for fix output_index
* server: remove unnecessary generated_tool_call_ids and oai_resp_tool_calls_started
* mtmd : read LFM2 tiling params from GGUF metadata
Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from
processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp
with the previous constants as defaults. do_image_splitting=false maps to
min_tiles=max_tiles=1 (no tiling).
Rebased version of #25524, original implementation by Michael DM (@michael-dm).
Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
* fix test
---------
Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
* metal : optimize argmax and pool_1d
Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : add pool_1d perf cases
Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : minor fix
* model : drop the redundant precision overrides in the Kolibri-1 router
Follow-up to the review in #30673: the router matmul works with the
default precision, so both ggml_prec_set_* calls can go away.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* cont : fix comment
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* opencl: fix OOB read in `mul_mv_mxfp4_f32_flat`
* opencl: remove large-N guard that forces CPU weight placement
* this condition claims that large batch produces incorrect results on
adreno and cannot be reproduced
* model loader uses N512 to probe op support, resulting in cpu fallback
* thus this condition does not seem necessary
* opencl: cap num of subgroups by workgroup size for q4_0 mc3
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add small batch variant for q4_k dp4a GEMM
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add a narrow variant of mul_mm_f16_f32_l4_lm
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add a one subgroup variant for gemv_noshuffle_q1_0_f32 for big M
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
---------
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
The __ARM_NEON branch defines both GGML_CPU_COMPUTE_* macros but selects
only FP16_TO_FP32, so FP32_TO_FP16 falls through to the portable
software conversion while neon_compute_fp32_to_fp16, a single fcvt,
stays defined and referenced by nothing. __POWER9_VECTOR__ and
__riscv_zfhmin select both.
Add an #error pair that refuses a branch defining a COMPUTE macro
without selecting it. That is how this stayed unnoticed: the fallback
silently supplies a correct software implementation, so the only
symptom is a hardware instruction nothing ever reaches. The
FP16_TO_FP32 half carries a !defined(__F16C__) carve-out, because PR
30343 keeps that direction on the lookup table on purpose.
Replace the comment above the fallback: it singled out NEON and POWER9
and pointed at a selection that now happens in the branch itself.
Hrishith Thadicherla