mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-10-28 17:56:13 +08:00
budiony
d81d0d8fb9
* Multi-column on Intel XMX matrix engines refactored Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0). * Fixed regression bug real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4 * Removed GGML_TYPE_Q8_0 activation * rewritten xmx_mul_mat 64-byte alignment and 16-byte pitch check on the activations * Implemented fused gate/up kernel One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types. * Removed the temporary XMX_NOGLU environment switch * Q2_K min-sum optimization Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group * Added dispatch check and GRF sweep tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%). Q6_K is unchanged. MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361. Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without. * Renamed variables * Added 2 environment variables GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it. * Update esimd.hpp 1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces. 2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once. The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.