darioooooo0o/Spark-X2.5-4B-GGUF

Spark-X2.5-4B Community Quants (imatrix GGUF)


Requests, questions or suggestions? Message me on X: https://x.com/imdariotoo

Community GGUF quantizations of XHToken/Spark-X2.5-4B, a 4.1B hybrid-attention (3:1 sliding-window:full) model with native 1M context. Quantized with an importance matrix calibrated on multilingual text, targeting fully GPU-resident inference on 4/8/12/16GB cards with at least 64K context (128K preferred).

Why the lineups look the way they do

Spark-X2.5-4B puts 69% of its parameters in FFN weights, and measured perplexity shows those FFN weights are uniformly sensitive: a layer-aware mix that drops ffn_up/ffn_gate to Q4K while keeping everything else at Q6K+ measured +0.76 PPL vs uniform Q6K. Layer-aware mixing only pays off when the VRAM budget forces tensors below Q5K - so the 12GB and 16GB quants are uniform (the headroom goes to context instead), while the 8GB and 4GB quants use mixes that protect the most sensitive tensors.

Available quants

*fit = weights + q4_0 KV + ~1.1 GB CUDA/compute overhead, one slot, -ngl 99, flash attention on.

8GB mix allocation (Q3KM-8GB-mix)

4GB mix allocation (Q4KM-4GB-mix)

Measured quality (perplexity, held-out multilingual corpus, 512 ctx, CUDA, RTX 3060 12GB)

For comparison, rejected experiments: a 5.53 bpw mix (attn Q6K / ffndown Q5K / ffnup+gate Q4K / embd Q80) measured 21.85 - worse than uniform Q6K despite a smaller file. Any allocation that puts Q4K or lower on ffn_up/ffn_gate while room remains for uniform Q5_K+ is a net loss on this model.

Context and KV cache

This is a hybrid model: only 9 of 36 layers carry full-attention KV (4 KV heads x 256 head dim). At q4_0 KV quantization that is ~10.1 KB/token:

• —64K: 0.68 GB | 128K: 1.36 GB | 256K: 2.72 GB | 512K: 5.44 GB
• —1M native context does NOT fit on any consumer card listed here (1M KV alone = 10.9 GB)

Recommended launch (llama.cpp)

Requires the vendor llama.cpp fork XHToken/llama.cpp (the spark2_5 architecture is not in upstream llama.cpp yet).

llama-server \
  -m Spark-X2.5-4B-Q4_K_M-4GB-mix.gguf \
  --chat-template-file chat_template.jinja \
  -c 131072 -ngl 99 -fa auto \
  -ctk q4_0 -ctv q4_0 -ctkd q4_0 -ctvd q4_0 \
  --temp 1.0 --top-p 0.95 --top-k -1 \
  --jinja --reasoning-format deepseek

Sampling per the model card: temperature 1.0, topp 0.95, topk -1. Thinking mode is on by default (qwen3-style reasoning); tool calls use the <tool_call>/<arg_key>/<arg_value> format and work through llama.cpp's auto tool-call parser.

Quantization method

1. 1.Importance matrix computed with llama-imatrix on 200 chunks of multilingual text (6 MB corpus, 512-token chunks) from the Q8_0 GGUF.
2. 2.Uniform quants: single Q6_K / Q5_K_M pass with --imatrix.
3. 3.Mixes: per-tensor type assignment via --tensor-type-file (290 tensors). ffn_down is kept at higher precision than ffn_up/ffn_gate in every mix - it is the KL-sensitive FFN tensor.
4. 4.Each variant validated with llama-perplexity on the same held-out corpus, sequentially on one RTX 3060 12GB, VRAM freed between runs.

Speed (RTX 3060 12GB, llama-bench, p512/n128, -ngl 99, -fa on, f16 KV, 3 reps)

At 40-60K context decode drops to 55-78 tok/s (long-context penalty); prefill holds 2500-3200 tok/s.

Agentic tool-use reliability (HermesBench subset, RTX 3060 12GB, 64K ctx)

Each quant ran a 6-task agentic probe subset (file/terminal, memory, todo, codeexecution, sessionsearch, cronjob tool classes) through a real agent harness, graded on both task correctness AND use of the required tool classes. Zero false-dones and zero timeouts on all quants. One harness-broken task excluded (failed identically on every quant including near-lossless Q6_K).

Tool-requirement compliance is stochastic at temp 1.0: repeated runs of the same task on the same quant flip between using and skipping the required specialized tool. Across 3 runs of 2 decisive tasks per quant:

Key observations:

• —Skipping the required tool is run-to-run variance for every quant, not a hard defect - even near-lossless Q6_K missed once. The mixes miss more often (50-67% vs 17%), so the quant does shift the odds, but any single run's pass/fail is not a reliable signal.
• —Raw task competence is intact everywhere: in every miss on every quant, the task was still completed correctly (right answer, right file) using generic tools instead of the required specialized one. Quant damage at these bitrates degrades tool-selection discipline, not capability.
• —Practical guidance: if your agent harness enforces tool-class requirements strictly, expect the mixes to fail such checks 3-4x more often than the uniform quants. If you only care about task outcomes, all four quants complete the work.

Files

License

Apache 2.0, matching the base model. Quantizations are derivative works of XHToken/Spark-X2.5-4B.