* server: include output_index in Responses API SSE events
* server: reuse Anthropic streaming code for fix output_index
* server: remove unnecessary generated_tool_call_ids and oai_resp_tool_calls_started
* mtmd : read LFM2 tiling params from GGUF metadata
Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from
processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp
with the previous constants as defaults. do_image_splitting=false maps to
min_tiles=max_tiles=1 (no tiling).
Rebased version of #25524, original implementation by Michael DM (@michael-dm).
Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
* fix test
---------
Co-authored-by: Michael DM <usable_siren180@simplelogin.com>
* metal : optimize argmax and pool_1d
Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : add pool_1d perf cases
Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : minor fix
* model : drop the redundant precision overrides in the Kolibri-1 router
Follow-up to the review in #30673: the router matmul works with the
default precision, so both ggml_prec_set_* calls can go away.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* cont : fix comment
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* opencl: fix OOB read in `mul_mv_mxfp4_f32_flat`
* opencl: remove large-N guard that forces CPU weight placement
* this condition claims that large batch produces incorrect results on
adreno and cannot be reproduced
* model loader uses N512 to probe op support, resulting in cpu fallback
* thus this condition does not seem necessary
* opencl: cap num of subgroups by workgroup size for q4_0 mc3
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add small batch variant for q4_k dp4a GEMM
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add a narrow variant of mul_mm_f16_f32_l4_lm
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add a one subgroup variant for gemv_noshuffle_q1_0_f32 for big M
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
---------
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
The __ARM_NEON branch defines both GGML_CPU_COMPUTE_* macros but selects
only FP16_TO_FP32, so FP32_TO_FP16 falls through to the portable
software conversion while neon_compute_fp32_to_fp16, a single fcvt,
stays defined and referenced by nothing. __POWER9_VECTOR__ and
__riscv_zfhmin select both.
Add an #error pair that refuses a branch defining a COMPUTE macro
without selecting it. That is how this stayed unnoticed: the fallback
silently supplies a correct software implementation, so the only
symptom is a hardware instruction nothing ever reaches. The
FP16_TO_FP32 half carries a !defined(__F16C__) carve-out, because PR
30343 keeps that direction on the lookup table on purpose.
Replace the comment above the fallback: it singled out NEON and POWER9
and pointed at a selection that now happens in the branch itself.
* spec : save the dynamic ngram cache file
* save the cache at the common_speculative_impl_ngram_cache destruction
* remove save_static/save_dynamic flags as well as the relevant TODO
* if the dynamic cache file does not exist, start with an empty cache
that is written on exit. if it exists but cannot be read, abort
instead of overwriting it.
* spec : fix draft_one not updating the ngram cache history correctly
* common_ngram_cache_update received only the new tokens with none of
the tokens before them causing ngrams ending at the new token to not
being formed, leading the context cache to only ever hold the prompt
* fix: pass the full history to common_ngram_cache_update with nnew set
to the number of new tokens the way llama-lookup does and also reset
cache_size to 0 on each new request
---------
Co-authored-by: Petros Sideris <petros.sideris@nokia.com>
* server : pass id_slot through the Anthropic /v1/messages conversion
server_chat_convert_anthropic_to_oai copies a small whitelist of common
params from the Anthropic request into the internal OAI-style body and
left id_slot out. Every other completion endpoint honours id_slot; a
proxy that pins a session to the slot holding its KV cache needs the
same on /v1/messages, or it silently falls back to automatic slot
selection and, on hybrid/recurrent models, pays a full re-prefill.
Adds a test asserting a request with id_slot set is served by that
slot, observed via GET /slots.
* server : shorten comments
* rm test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* tests : make MoE router selection independent of the backend noise
The router scores are in the range [0, 1), so a random draw can leave the
last expert (or expert group) still selected and the first one dropped tied
within the noise between the backends. They then route a token to different
experts and the logits differ by 4e-3 instead of the usual 5e-7.
Beam is the only arch here with a real top-k (8 experts, 2 used) and a group
top-k (4 groups, 2 used), so it is the only one that hits this: 2 of 20 seeds
failed on an M2 Ultra, the other MoE archs select 2 of 2 and cannot tie.
Fill the selection bias with a ramp of step 2 to give every expert a score
range of its own, which turns the top-k into a function of the expert index
alone. The bias affects only the selection, the weights still come from the
unbiased probs, so the MoE output keeps its magnitude. Beam now matches the
CPU reference within 4e-7 for all of 160 seeds.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : tell the CPU and the GPU of an Apple SoC apart in the device table
Both devices report the same description ("Apple M2 Ultra" here), so the
device column of the test table had two indistinguishable rows and a failing
row did not say whether the Metal device or the host CPU produced it.
Add the device name, which is also what -b takes, unless the description
already is the name.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* hex-build: bump toolchain, enable v85
* hex-build: bump toolchain to v0.8 in the CI and build scripts
* hex-build: add a little python script for swiv needed on some iot devices
* hex-build: fix flake8 warnings
* hex-arch: update version cap to v85
* hex-inspect: flexible TOOLS_ROOT
* hex-build: update to snake_case naming
* hex-build: update leftover old toolchain refs
* hex-target: update build and run scripts to support ubuntu with adb transport
SearchInput already renders its own clear button.
Hide the browser's native search cancel button to avoid showing both controls.
Fixes#28627
Signed-off-by: aiqubits <aiqubits@hotmail.com>
* init
* wip
* wire it up
* make it clear that it's experimental
* add props.connect_code
* nits
* fix windows build
* server: do not advertise a connect code without the tunnel
--connect-code on its own had no effect other than being served in
/props, so the Web UI showed a remote access code that no tunnel was
serving. Clear it when --connect is off, and say so once at startup.
The teardown comment claimed that terminate() closes the child stdout.
It does not: the log thread leaves fgets() only once every writer of
that pipe is gone, so a descendant of llama-connect would block the
join forever. Describe the real contract instead.
Also restore the --reasoning-preserve default in the generated docs.
* Update tools/connect/CMakeLists.txt
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
* server: dial llama-connect on the bound port and a reachable host
Pass ctx_http.port so --port 0 and the GCP override work, pick the
dial host from --host (wildcards map to loopback, IPv6 is bracketed)
and refuse --connect when only unix sockets are bound.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
In the attachment preview, SyntaxHighlightedCode's wrapper (overflow-auto) was not the element that ended up owning horizontal overflow: the inner code box kept its content clipped at the wrapper's width, so a single line longer than the preview pane was cut off with no scrollbar (scrollWidth == clientWidth while the code's own content measured tens of thousands of px).
Give the pre/code boxes their intrinsic width (width: max-content, min-width: 100%) so long-line overflow propagates to the scrolling wrapper. Verified in the running WebUI: with a 6000-char line, wrapper scrollWidth goes from 990 (clipped, unscrollable) to 50449 with a working horizontal scrollbar; short content still fills the container exactly as before (min-width: 100%).
Fixes#30796
The fused moe_reduce kernel computes dst = sum(experts[e] * weights[e]) in a
single dispatch, reading experts and weights while writing dst. The graph
allocator computes tensor liveness on the unfused graph, where the inputs are
fully consumed by earlier nodes of the fused group, so it can legally recycle
the input memory as dst. In the fused kernel this becomes a
threadgroup-vs-threadgroup race that produces rare non-deterministic wrong
results (observed with test-llama-archs -a kolibri on the meta backend).
Reject the fusion when dst overlaps either input, same guard as the
mul_mat_add fusion. The unfused MUL + ADD path is always correct.
Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
* cuda: cap Stream-K blocks at the number of k-blocks
Stream-K normally launches one CUDA block per SM. For small matrices,
this can exceed the total number of output-tile/k-block pairs, leaving
blocks with no work and adding launch and fixup overhead. Cap the grid
at that count to remove these excess blocks without changing the useful
rounded K ranges or numerical results.
Assisted-by: Codex
* cuda: cap Stream-K blocks at the number of k iterations
Cap the grid at the number of K_vram iterations across all output tiles, rounding up partial iterations. This removes extra CUDA blocks left by the k-block cap.
Partial iterations can change FP32 summation grouping. MUL_MAT and MUL_MAT_ID backend tests pass on RTX 5090.
* Update ggml/src/ggml-cuda/mmq.cuh
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ui : keep markdown tables inside the chat column on phones
The phone rule made the table wrapper full-bleed with -1rem margins, so the
first and the last column sat under the screen edge. Keep the wrapper inside the
chat column and cap it at its width, so a wide table scrolls in place instead.
Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* test : show a table-only model answer in the chat message story
Adds a story with an answer that is mostly a table, so the table layout can be
checked at a phone width.
Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* ui : indent markdown lists by 2rem
1.5rem put the marker too close to the item text.
Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* The problem here in the repo is gguf-split can silently produce corrupted tensor output when the input GGUF file is truncated. The fix is to add tensor-data bounds checks and short-read checks in gguf-split.cpp and export-lora.cpp, and a regression test that checks truncated input is rejected and valid tensor data is preserved.#
* Bug : GGUF-split can silently produce corrupted tensor output when the input GGUF file is truncated. Fix : add short-read checks in gguf-split.cpp (in copy_file_to_file in split_strategy) and export-lora.cpp (in read_tensor_data of file_input) so that truncated input is rejected.
* tests: add beam fusion rows to MTL baseline
Regenerate tests/fusion/MTL.csv with test-fusion --record. The newly
added beam architecture has no baseline rows, so test-fusion fails with
2 failing rows (MUL+ADD x6, RMS_NORM+MUL x25). All other arch counts are
unchanged.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* skills: add update-fusion-table skill
Add a skill for regenerating a per-device test-fusion baseline. It keeps
the workflow from tests/fusion/README.md and the parts that are easy to
get wrong: diff the recorded file before applying it, the distinct
meanings of a FAIL row, and never recording a partial run.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
.pi/gg/SYSTEM.md repeated the full agent ruleset. Move it to .pi/gg/APPEND_SYSTEM.md and keep only the llama.cpp-specific rules, so it can be appended to a main system prompt with --append-system-prompt.
Drop the .pi/SYSTEM.md symlink setup from wc2wt.sh and pr2wt.sh. Worktrees no longer need it now that the folder holds an addendum instead of a full prompt.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* CUDA: fix KQ write-after-read race in flash_attn_ext_vec
Assisted-by: ChatGPT
* CUDA: apply review suggestions for KQ sync comments
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* hex-fa: support for masked block skipping to reduce DMA fetching
* hex-fa: finetuning the hmx->hvx crossover thresholds (boosting gemma-4)
* hex-fa: update hmx gate to account for mdev.count
Port of the Vulkan HC_POST_GATE fusion (#29520). hc_combine builds the
hc_post weights as 2*sigmoid(inject/hc); dsv4_hc_post_f32 now takes a
gated template flag and computes scale_out*sigmoid(scale_in*p) from the
raw weights, so the chain runs as one kernel instead of four.
* ggml: support non-contiguous SUM (CPU, CUDA)
* tests: add non-contiguous SUM cases
* tests: SUM over non-contiguous views
* vulkan: support non-contiguous SUM
* webgpu: support non-contiguous SUM
Renzis