Ttimms/Spark-X2.5-4B-NVFP4

Spark-X2.5-4B — NVFP4 W4A16 (GPTQ)

NVFP4 weight-only (W4A16) quantization of [`XHToken/Spark-X2.5-4B`](https://huggingface.co/XHToken/Spark-X2.5-4B) — 4-bit float weights in 16-element blocks with FP8 block scales, activations in bf16. lm_head and every self_attn.g_proj (the head-wise attention-output gate) kept in bf16, matching XHToken's own INT8 exclusion set. Rounding is GPTQ (Hessian-aware), not RTN — this matters a lot here (see the eval). Produced with llm-compressor 0.13.

• —2.7 GiB on disk (bf16 base is 7.7 GiB — ~65 % smaller)
• —The first NVFP4 / 4-bit compressed-tensors build of Spark-X2.5-4B
• —Serves on vLLM on Blackwell via the out-of-tree Spark2_5 plugin (below); on SM120 the weight-only path decodes through the Marlin kernel to a bf16 GEMM (no native FP4 compute), so the benefit here is footprint, not raw speed
• —Leaves ~10 GiB of KV headroom on a 16 GB card — room for a large slice of the 1 M-token context window

Evaluation

lm-evaluation-harness, vLLM 0.26 backend (+ Spark25 plugin), greedy decoding, **median of 3 draws** with the range (the harness is non-deterministic run-to-run even at greedy — a single draw is not a reproducible score). HumanEval-instruct `pass@1` / `createtest` (n = 164); MBPP base 3-shot (n = 500). RTX 5070 Ti (Blackwell, SM120).

GPTQ rounding is doing real work on this model — plain RTN loses 8.5 pp on HumanEval and collapses on MBPP (−15.6 pp, 33 %). GPTQ recovers +4.3 pp / +13.6 pp, landing at −4.3 pp HumanEval and −2.0 pp MBPP (the MBPP Wilson intervals overlap the bf16 baseline). For a 4-bit build at 65 % smaller, that is a usable trade. If you want maximum quality retention, XHToken's own FP8 build is the safer choice.

Serving (vLLM + out-of-tree plugin)

Spark2_5ForCausalLM is not in the vLLM model registry. Install the general plugin into the same environment as vLLM:

git clone https://github.com/XHToken/Spark-plugin
pip install -e ./Spark-plugin --no-deps
pip install -U "openai>=2.25.0"

Then serve:

vllm serve Ttimms/Spark-X2.5-4B-NVFP4 \
  --trust-remote-code \
  --enforce-eager \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --enable-auto-tool-choice --tool-call-parser spark25

On WSL, also set VLLM_USE_V2_MODEL_RUNNER=0 (the V2 runner needs UVA, which WSL disables). --enforce-eager is the safe default on SM120; weight-only NVFP4-W4A16 also runs with CUDA graphs.

Method & provenance

• —Quantizer: llm-compressor 0.13, GPTQModifier(scheme="NVFP4A16", targets="Linear", dampening_frac=0.1), ignore lm_head + re:.*self_attn\.g_proj + tied embeddings. Calibration: 512 samples of theblackcat102/evol-codealpaca-v1 (code, disjoint from the eval sets), 2048 tok.
• —Base: `XHToken/Spark-X2.5-4B` (Spark2_5ForCausalLM, ~4.1 B, hybrid sliding/full attention, head-wise attention-output gate, 1 M context, Apache-2.0).
• —Built and evaluated on an RTX 5070 Ti (Blackwell, SM120), vLLM 0.26 + the Spark2_5 plugin.

License

Apache-2.0, inherited from XHToken/Spark-X2.5-4B.