alesha-pro/DeepSeek-V4-Flash-0731-REAP-K160-GGUF

DeepSeek-V4-Flash-0731 REAP K160, Q3K/Q4K GGUF

A GGUF quant of `0xSero/DeepSeek-V4-Flash-0731-REAP`, the K160 expert-pruned checkpoint of `deepseek-ai/DeepSeek-V4-Flash-0731`. The pruning is 0xSero's work: 160 of 256 routed experts kept per MoE scope, top-6 routing unchanged. I only did the quantization and the testing below.

I built this to fit a 4x RTX 3090 rig (96 GB) with a 256K context window. The file is 89.9 GB, which leaves enough room on four 24 GB cards for a quarter million tokens of KV cache.

What is inside

Routed experts carry the Q3K and Q4K bulk. Attention, shared experts, the indexer and the output head stay at Q8_0. No imatrix was used.

180.4B parameters after pruning, 89.92 GB of tensor data, so about 3.99 bits per weight.

Quality: a tie with a 2-bit quant of the unpruned model

I ran this against DeepSeek-V4-Flash-0731-UD-IQ2_M, which keeps all 284B parameters at 2.56 bpw and weighs almost exactly the same, 90.9 GB. Both served from llama.cpp at temperature 0, 150 scenarios total.

One scenario out of 150 is noise, so I read this as a tie. The two builds do not fail on the same items though: K160 won CLI-16/17/28/30/40 and HA-17/19, the 2-bit build won CLI-01/03/11/35 and HA-08/13. Same level, slightly different error profile.

Which is the interesting part. Same 90 GB budget, and it does not matter much whether you spend it on 180B parameters at 4 bits or 284B parameters at 2.5 bits.

Known problem: non-English output

Russian output from this build is broken. Not degraded, broken. The unpruned 2-bit quant handles the same prompts fine, so this comes from the pruning, not from my quantization. My guess is that REAP drops the experts that rarely fire on an English and code calibration set, and other languages live exactly there.

I only tested Russian. If you need any language other than English, test it before you commit to this file, or use a quant of the unpruned model.

Running it

This needs my llama.cpp fork (`alesha-pro/llama.cpp`, branch ds4-longctx) built with CUDA. Mainline llama.cpp cannot load DeepSeek V4 Flash at all.

git clone -b ds4-longctx https://github.com/alesha-pro/llama.cpp
cd llama.cpp
cmake -B build-v4-cuda -DGGML_CUDA=ON && cmake --build build-v4-cuda -j

MODEL=/path/to/DeepSeek-V4-Flash-0731-REAP-K160-Q3_K_Q4_K-00001-of-00004.gguf \
CTX=262144 bash scripts/ds4-prod-serve.sh

scripts/ds4-prod-serve.sh carries the whole ship flag set, so there is no wall of DSV4_* variables to copy. Every one of them is still overridable from the environment if you want to A/B a flag, and the kill switches are DSV4_PREFILL_GRAPHS=0 and GGML_CUDA_DISABLE_GRAPHS=1.

The launcher also warms the server to full depth once at startup, which takes four to five minutes. That matters more than it sounds: without it the first request at any new depth pays a one-time allocator climb and runs roughly 3x slower than steady state. With it, even the first real request runs warm.

Point MODEL at the first shard, llama.cpp picks up the rest. TS defaults to the tensor split for my four cards and you will probably need your own. The chat template ships inside the GGUF.

Speed on my rig

Four RTX 3090 on PCIe 3.0 x16, all four at a 350 W power limit, the fork above, 262144 context, weights and cache filling 92 of the 96 GB.

Prefill runs 1675 t/s on a 32K prompt and 1766 to 1771 t/s between 64K and 130K. Marginal prefill, measured after the fixed cost is paid, sits around 1830 to 1850 t/s. Short prompts look slower than that: a real 18K request today showed 1660 t/s apparent, because every request pays a fixed cost of about 3 seconds for the context checkpoint, and on a short prompt that cost dominates the average.

Decode is 34 to 36 t/s in the benchmark harness and up to 38.9 t/s on single requests.

If you run 3090s at their stock limit, expect less. On other models the step from 220 W to 350 W is worth a lot on the prefill side, so compare against your own cards at a known limit rather than against these numbers.

Credits

Expert pruning and the REAP checkpoint: 0xSero. Observation dataset behind the expert ranking: `0xSero/deepseek-v4-flash-reap-observations-v2`. Base model: DeepSeek. Quantization, benchmarks and the llama.cpp work: me, @superalesha.