jmarceno/Xing4.0-29B-A4B-GGUF

Xing4.0-29B-A4B — dynamic 3.31 BPW GGUF (+ the imatrix used to build it)

A 12.04 GiB / 3.31 BPW dynamic quantization of Xing4.0-29B-A4B (China Telecom / TeleAI, Apache-2.0), plus the importance matrix and the exact recipe used to make it — so the quant is reproducible and re-tunable.

Run it on [jmarceno/llama.cpp-xing4](https://github.com/jmarceno/llama.cpp-xing4) (branch xing4_0-port). That fork is the Xing4.0 engine plus CUDA changes that keep q80/q40 MLA K/V quantized inside FlashAttention and keep 64-expert MoE fusion valid. The stock `shuxiaoqiong/llama.cpp` `xing4_0-port` binary loads this file, and it converts the whole KV cache to f16 on every attention step. Use the fork for speed.

This repo exists because the upstream publisher ships one IQ4_NL artifact and does not publish the imatrix behind it. An imatrix cannot be recovered from a quantized GGUF (it is activation statistics, not weights), so we regenerated one from the BF16 source and built a smaller artifact. Both are here.

Files

sha256 (quant): d33dd4172959af2f55c458b8168ac7cc9a90a26e9e68520653ec61fc50836e75

Recipe

Base ftype Q8_0, with expert families overridden:

The down-projection gets more bits than gate/up because its output feeds the residual stream unattenuated — the same family-level logic measured in docs/qwen36-mix-stq-experiment.md.

Measured

Same engine, same flags, held-out WikiText-2 raw test (-c 512 --chunks 16, English-only corpus — a caveat, the model is Chinese-centric):

Speed on 2× RTX 3060 (CUDA, all layers offloaded): pp512 1100.6 t/s, tg128 65.9 t/s; ~2.8× faster than the upstream IQ4_NL at PPL preprocessing (2.70 s vs 7.46 s per 512-token chunk). Generation through llama-server: 52.9 t/s.

Net: 36 % smaller for +19.9 % perplexity.

Running it

xing4_0 (mHC + MLA + 64-expert MoE + MTP block) is not merged upstream yet. Upstream support is open PR ggml-org/llama.cpp#29012.

Build [jmarceno/llama.cpp-xing4](https://github.com/jmarceno/llama.cpp-xing4), branch `xing4_0-port`. It is the vendor port plus:

• —in-kernel dequant of q80/q40 K/V in the MMA FlashAttention kernel for absorbed MLA (K=576, V=512, V a view of K), so the graph does not convert or reserve a full-context f16 KV scratch (GGML_CUDA_FA_INKERNEL_DEQUANT=0 restores the old path);
• —CUDA-graph allocator edges for top-k MoE fusion (upstream llama.cpp#28432), so a valid 64-expert fusion is not rejected.

A 2048-token greedy smoke matched the previous binary on the same prompt. The removed conversion grows with context; that is why the fork is the one to serve with. Daily KV for this quant is q80/q80, which is the shape the kernel handles.

# build https://github.com/jmarceno/llama.cpp-xing4 branch xing4_0-port, then:
./llama-server \
  -m Xing4.0-29B-A4B-DYN-DIQ4XS-GU2XXS-Q8_0.gguf \
  --alias xing4-29b-a4b -c 131072 --parallel 1 \
  --n-gpu-layers 99 -ctk q8_0 -ctv q8_0 -fa on \
  --jinja --chat-template-kwargs '{"enable_thinking":true}' \
  --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
  --spec-type none

Sampling follows the model card: temp 1.0 / top_p 0.95 / repetition_penalty 1.05 for complex reasoning and general tasks, temp 0.8 for coding/agent tasks. The card specifies no top_k, so leave it disabled. Measured VRAM on 2× 12 GB (KV q80/q80): 65536 ctx → 5.2 GiB free on the display card; 131072 → 4.5 GiB; 262144 (native max) → 3.1 GiB, but only ~305 MiB free on the second card, so it is tight for long prefills.

Reproducing the quant

# 1. BF16 GGUF from the official safetensors
python3 convert_hf_to_gguf.py /path/to/Xing4.0-29B-A4B \
  --outfile Xing4.0-29B-A4B-BF16.gguf --outtype bf16

# 2. imatrix (this is the published file; collected on an IQ4_NL working model
#    because the 62 GB BF16 model does not fit 24 GB VRAM + 31 GB RAM)
llama-imatrix -m Xing4.0-29B-A4B-IQ4_NL-working.gguf -f calibration.txt \
  -o imatrix-Xing4.0-29B-A4B-820chunks.gguf -c 512 -ngl 999

# 3. quantize with the recipe (see quantize-recipe.txt for the full list)
llama-quantize --imatrix imatrix-Xing4.0-29B-A4B-820chunks.gguf \
  --output-tensor-type q6_k --token-embedding-type q8_0 \
  --tensor-type 'ffn_down_exps\.weight=iq4_xs' \
  --tensor-type 'ffn_gate_exps\.weight=iq2_xxs' \
  --tensor-type 'ffn_up_exps\.weight=iq2_xxs' \
  ... Xing4.0-29B-A4B-BF16.gguf out.gguf Q8_0

The calibration corpus was assembled round-robin from WikiText-2 raw, a source tree, and Chinese Wikipedia extracts (12.8 MB total; 820 chunks were consumed, ≈420K tokens). The imatrix is reusable: llama-quantize --imatrix accepts it directly for any other Xing4.0-29B-A4B recipe (IQ3_XXS, Q4 mixes, ...).

Notes / caveats

• —The MTP (`blk.40`) experts are pinned to IQ4_XS / IQ4_NL because llama-imatrix was collected with the block unloaded and has no entries for it. llama-quantize refuses imatrix-requiring types (IQ1*/IQ2/IQ3_XXS) on those tensors. The upstream IQ4_NL artifact made the same choice (592 imatrix entries). On this fork, `blk.40` runs when you pass `--spec-type draft-mtp`; with speculation off the loader prints `unused tensor blk.40.` and skips it.
• —Perplexity was measured on an English-only corpus while the model is Chinese-centric; treat the numbers as a relative A/B between the two quants.
• —The engine's llama-cli has no single-turn mode and loops on stdin EOF; use llama-server for generation tests.

Quantized and measured 2026-09-17 on 2× RTX 3060 12 GB. Base model: Xing4.0-29B-A4B, Apache-2.0, © China Telecom AI Technology Co., Ltd.