Myric/Spark-X2.5-4B-GGUF

Spark-X2.5-4B — GGUF

Full-precision GGUF conversion of `XHToken/Spark-X2.5-4B`.

⚠️ Requires a patched llama.cpp — upstream will NOT load this

spark2_5 is not in upstream llama.cpp. Loading this file with a stock build fails at the architecture check. Build from the branch that adds it:

git clone -b model/Spark2_5 https://github.com/brywil/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j
./build/bin/llama-cli -m Spark-X2.5-4B-BF16.gguf --ctx-size 32768 -ngl 999 -p "..."

Based on upstream 6d0549831. The branch adds the arch, a converter, and a new BPE pre-tokenizer type; nothing else is changed.

Always pass `--ctx-size`. The model declares max_position_embeddings: 1048576, and llama.cpp defaults to the trained context — that will try to allocate the full 1 M window.

What's here

All 362 tensors present in every file, attn_gate F32 in all of them, and zero block-32 substitutions — verified by reading the type histogram out of the shipped bytes rather than trusting the filename.

The bf16 is published deliberately: the arch needed a fork to run at all, so it is the reference artifact to check a quant against.

Which one

Q5_K_M is the default recommendation — 2.75 GiB fits an 8 GB card with room for real context, and 128k of KV costs only another 4.5 GiB. Q4_K_M if you are tight on VRAM. Q8_0 if you want a near-reference artifact without carrying 7.67 GiB.

The imatrix was built from a code-inclusive mixed corpus, 1,000 chunks × 512 tokens = 512k tokens, chosen over a prose-only set because the intended workload is agentic coding.

The model

4.11 B dense, 36 layers, apache-2.0. Interleaved sliding-window attention — 27 sliding (window 512) and 9 full, in a strict sssF pattern. 16 query heads, 4 KV heads, head_dim 256, hidden 2560, GELU FFN, 131,072 vocab, tied embeddings. It is a reasoning model and emits a thinking block.

Long context is cheap here, by design

Only the 9 full-attention layers scale with context; the 27 sliding layers are bounded by their 512 window.

9 layers x 4 KV heads x 256 head_dim x 2 (K+V) x 2 bytes = 36 KiB/token

So 128k of context costs less KV than the weights do. Note this is a memory calculation, not a retrieval claim — long-context quality is unmeasured here.

Notes for anyone quantizing this

• —Every dimension is divisible by 256 (hidden 2560, intermediate 10240, vocab 131072, KV dim 1024), so all QK_K types are legal. A file labelled Q4KM really will be Q4KM — unlike archs with an awkward dimension, where llama-quantize silently substitutes block-32 types.
• —Pin `attn_gate` to F32. g_proj is Linear(2560, 16) — 40,960 params per layer, 1.47 M total, about 5.9 MB. Its sigmoid output multiplies the attention result, so error there compounds along the sequence instead of adding to one layer's output, which is exactly the case where per-tensor weight error under-reports the true cost. At this size the bits are free. The converter on the branch above emits it as F32 — but `llama-quantize` will still quantize it unasked, because ne0 is 2560 and therefore 256-divisible, so Q5_K is perfectly legal on [2560,16]. Legal is not correct here. Pass it explicitly:

  ./build/bin/llama-quantize --imatrix spark-x25-4b.imatrix --tensor-type attn_gate=f32 \
      Spark-X2.5-4B-BF16.gguf out.gguf Q5_K_M

That is how every quant in this repo was built. Cost: 5.9 MB on a 2.7 GiB file.

Verification

Greedy generation from this file is byte-identical to the HF reference (transformers 4.57.1) over 24 tokens on "The capital of France is Paris. The capital of Japan is" — a span crossing six full-attention and eighteen sliding layers, so both rope configurations and the attention gate are exercised. That check matters more than usual on this arch: it uses two different rope setups selected per layer type (sliding layers rotate 256 of 256 dims at theta 10,000; full-attention layers rotate only 64 of 256 at theta 5,000,000), and applying one rope to every layer yields a model that loads and produces fluent, wrong output with no error anywhere.

Measured

Throughput (GB10 / DGX Spark, unified memory, ~123 GB/s achieved)

Do not compute a ratio from those two rows — they were served under different slot counts, and the bf16 figure is per-slot under 4-way batching (~74 t/s aggregate). The clean statement is that decode here is bandwidth-bound: 2.75 GiB of weights at ~123 GB/s predicts ~45 t/s, and Q5KM measures 48.3.

Agentic coding, 42 tasks (bf16)

Driven through a real opencode agentic loop with tool calling, graded by a fresh pytest per task — not a multiple-choice benchmark.

32 of 32 gradeable tasks passed. The 10 excluded tasks were harness limits, not wrong answers: 8 were cut off by a 32,768-token output cap and 2 by the wall clock, with no degeneration detected in any of them. They are reported as void rather than failed because the model was still working when the harness stopped it. Treat 32/32 as a floor on capability, not a measurement of it.

The output-token gradient (6,120 → 6,638 → 22,340) is the more informative number: cost stays flat from general to hard, then rises 3.6× at frontier, which locates this model's ceiling between those two tiers.

Do not use wikitext perplexity to pick a quant on this model

A full ladder on one binary, corpus and config (wiki.test.raw, 137 chunks) came out non-monotonic, with three quants scoring "better" than bf16:

The repeat pass is identical to four decimals, so the noise floor is zero and those differences are real measurements — they simply are not measuring quantization damage. Two structural reasons: the corpus is English prose against a model whose chat template and 131k vocab indicate Chinese-primary training, and 27 of the 36 layers see only a 512-token window, so 2048-token chunks exercise the long-range path in 9 layers and nothing else in the other 27. An English-prose PPL on this arch measures roughly a quarter of it.

Use KL divergence against the bf16 logits instead — it asks "how far did quantization move the model from its own full-precision self", which is the question that matters:

same top-1 % is the figure worth quoting. Q5KM and Q4KM are pending and will be added.

Provenance

Quantized with llama-quantize from the model/Spark2_5 branch at 498cbc7e0 (upstream base 6d0549831), x8664 CUDA build verified to contain the arch before use. Every file was loaded and generated from before publication — the PPL ladder above is that evidence — and Q5K_M has additionally been driven through the full 42-task agentic suite.

Credit

Model by XHToken. This repo is a format conversion only — no weights were retrained or altered.