puwaer/DeepSeek-V4-Flash-0731-reap-200b-gguf

puwaer/DeepSeek-V4-Flash-0731-reap-200b-gguf

GGUF builds of puwaer/DeepSeek-V4-Flash-0731-reap-200b for llama.cpp — 178 routed experts per layer (down from the base model's 256) via router-weighted expert activation pruning (REAP). See that repo's card for the compression recipe and full benchmark discussion; this one covers what is specific to the GGUF files: what each one is, how it was quantized, and how to serve it.

Produced with moe-compress and llama.cpp.

Files

`MXFP4_MOE` is not a quantization, it is the baseline. DeepSeek ships this model's routed experts already in MXFP4 (expert_dtype: fp4); llama.cpp's converter repacks those values into ggml's MXFP4 block layout without changing a single one (conversion/deepseek.py::_pack_mxfp4_blocks, 17 bytes per 32 values either way). So this file is numerically identical to the safetensors checkpoint linked above — same weights, different container.

Every rung below it is a requantization of already-4-bit data, not a quantization from bf16 (--allow-requantize is required), and costs somewhat more quality than the same rung would starting from a full-precision checkpoint. That is true of every public GGUF of this model, not a property of this pipeline.

Attention, the indexer and the shared experts stay Q8_0 at every rung; the entire size budget below MXFP4_MOE goes to the routed experts (ffn_{gate,down,up}_exps), which are ~90% of the checkpoint.

Quantization recipe

Built with a commit-pinned llama.cpp (`42e98813e`, the first commit with DeepSeek-V4 support) on Hopper (sm_90):

python3 convert_hf_to_gguf.py DeepSeek-V4-Flash-0731-reap-200b --outtype bf16 \
    --outfile DeepSeek-V4-Flash-0731-reap-200b-MXFP4_MOE.gguf

llama-imatrix -m DeepSeek-V4-Flash-0731-reap-200b-MXFP4_MOE.gguf \
    -f imatrix-calibration.txt -c 512 -ngl 99 \
    --output-format gguf -o imatrix.gguf

llama-quantize --allow-requantize --imatrix imatrix.gguf \
    --tensor-type-file recipes/q3km.txt \
    DeepSeek-V4-Flash-0731-reap-200b-MXFP4_MOE.gguf \
    DeepSeek-V4-Flash-0731-reap-200b-Q3_K_M.gguf Q3_K_M

(repeated per rung, swapping the recipe file and ftype: iq3xxs.txt/IQ3_XXS, q2k.txt/Q2_K, iq1m.txt/IQ1_M)

The imatrix is computed against this checkpoint's own 178 experts, not borrowed. The only public DeepSeek-V4-Flash-0731 imatrix (tarruda/DeepSeek-V4-Flash-0731-GGUF) is sized for the base model's 256 experts; llama-quantize checks imatrix_size == ne[0]*ne[2] and refuses a mismatch outright. The calibration corpus is that repo's own imatrix-calibration.txt, so the numbers stay comparable to a known-good reference even though the statistics are computed fresh, against this checkpoint's own kept experts.

Serving

llama-server -m DeepSeek-V4-Flash-0731-reap-200b-Q3_K_M.gguf \
    -ngl 99 --ctx-size 4096

The chat template is embedded in the GGUF (the same chat_template.jinja as the safetensors checkpoint — thinking on by default, see below), so no --chat-template flag is needed. Pass --reasoning-format deepseek to split the <think>...</think> block into message.reasoning_content instead of leaving it inline in message.content.

Tool calling works — pass --jinja and tools=[...] the standard OpenAI-compatible way; role: "tool" messages and the model's own tool_calls are both handled by the embedded template. (An earlier build of this GGUF raised unsupported role tool and had no tool-call parser at all; that's fixed as of this file's regeneration.)

Full GPU offload (`-ngl 99`) is what this pipeline verified against. Batched CUDA prefill of a partially-offloaded DeepSeek-V4 graph has been observed to fail (GGML_ASSERT in mul_mat_id) on checkpoints where a token's routing row names the same expert twice; this checkpoint has that fixed, but low -ngl was not re-tested after the fix.

This checkpoint is 103.85 GiB even at the smallest rung above — it does not fit one 96 GB card. -ngl below 99 with the rest on host RAM, or a multi-node rpc-server split, are both untested here.

Benchmarks

The scores below are the source checkpoint's, measured with SGLang on the safetensors weights — see puwaer/DeepSeek-V4-Flash-0731-reap-200b for the full discussion. They carry over exactly to `MXFP4_MOE`, which is bit-identical to that checkpoint. The quantized rungs (Q3_K_M and below) have not been benchmarked separately — expect some further accuracy cost, unmeasured here.

Difference from the base model, in points:

Metrics: GSM8K exact_match,strict-match, MATH-500 math_verify,none, HumanEval+/MBPP+ pass@1_plus. All greedy (n=1), 4096-token context, enable_thinking=false.

What changed relative to the base model

One difference deserves to be read before you deploy this:

• —The multi-token-prediction modules are gone. The base checkpoint carries mtp.0, mtp.1 and mtp.2; this one carries none of them. MTP-based speculative decoding is therefore unavailable. Engines that look for those weights will fall back to ordinary decoding. Nothing else references them, so standard generation is unaffected.

Everything else about the prompt and generation defaults is the base model's.

Thinking

Thinking is on by default, just like the base model.

# Python, against the safetensors checkpoint
tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False)

# llama-server / SGLang API
{"messages": [...], "chat_template_kwargs": {"enable_thinking": false}}

• —Reasoning effort: Set reasoning_effort to "low" (default), "high", or "max" (applies only in thinking mode).

Practical Notes

1. 1.Token Budget: Thinking consumes tokens before the actual answer starts. Ensure your context/output budget is high enough to prevent mid-reasoning truncation.
2. 2.Default Sampling: The safetensors checkpoint samples by default (do_sample, temperature, top_p) rather than using greedy decoding; set your own sampling params when serving the GGUF, since generation_config.json is not embedded in it.
3. 3.Output Format: Reasoning appears inline in message.content, ending with </think>, unless split out — with llama.cpp, --reasoning-format deepseek moves it to message.reasoning_content.

Choosing between REAP and REAM

On this model REAP wins outright, and by a margin that widens as more is removed. Points against the base model, given as 178 experts / 132 experts:

REAP at 178 experts returns the base model's pass@1 exactly on both code benchmarks — 542 problems, not one of them different — for a third off the checkpoint.

Note also how differently the two scale. Going from 178 experts to 132 costs REAP 1.1 more points of GSM8K; it costs REAM 17.0.

Citation

The methods:

• —REAP — Router-weighted Expert Activation Pruning. Lasby et al., 2025. arXiv:2510.13999
• —REAM — Router-weighted Expert Activation Merging. Jha et al., 2026. arXiv:2604.04356

The compression implementation: https://github.com/puwaer/moe-expert-compress

The GGUF conversion and quantization: https://github.com/ggml-org/llama.cpp

The base model: deepseek-ai/DeepSeek-V4-Flash-0731

License

MIT, following the base model. The compression code is MIT; its numerical core is ported from the official REAM reference implementation (Copyright (c) 2026 Samsung Electronics Co., Ltd.) with attribution headers retained. llama.cpp is MIT-licensed and imposes no further terms on its output.