sleepyeldrazi/DeepSeek-v4-Flash-REAP-K150-NVFP4

DeepSeek V4 Flash — REAP K150 Mixed NVFP4 (NVFP4 + Q2_K + Q8)

REAP-pruned DeepSeek V4 Flash at K150 (150 routed experts per MoE layer, 41% pruning), with a hybrid mixed-precision quantization targeting NVIDIA DGX Spark (GB10 Blackwell, sm_121).

At a Glance

Mixed Quantization Structure

This model uses a hybrid quant strategy optimized for DGX Spark's bandwidth profile (273 GB/s LPDDR5X, ~97 GB/s managed-memory serving path):

Why this mix?

The bottleneck in MoE decode is gate+up expert projection (~2/3 of expert bytes). Standard GGUF quants use IQ2XXS here. NVFP4 reads more bytes (4.50 vs 2.06 bpw) but offers substantially better precision at the same bit budget, with dequant bandwidth at ~140 GB/s on GB10. In practice, NVFP4 decode is slightly slower than IQ2XXS due to the higher byte volume, but the precision gain is well worth the small throughput cost. Down experts stay Q2K (already saturating at ~160 GB/s). Attention and shared experts stay Q80 for quality.

NVFP4: lossless MXFP4→NVFP4 conversion

The HF source checkpoint stores experts in MXFP4 (e2m1 nibbles + e8m0 per-32 block scale). NVFP4 is e2m1 + e4m3 per-16 + per-expert fp32 scale_2. The e2m1 nibbles are identical — the conversion is a scale-only transform. No weight requantization, no quality loss from the quantization step. The NVFP4 weights are bit-identical to the MXFP4 originals; only the scale format changes.

Domain Split (Calibration)

8,000 prompts · 5.0M tokens · calibration on DGX Spark (NVIDIA GB10) at 4,096 token context. REAP activation_energy_sum2 score metric.

How to Run

⚠️ Standard llama.cpp / GGUF runtimes will NOT load this model. It uses a custom NVFP4 multi-tensor GGUF convention (.nvfp4_weight + .nvfp4_scale_2) plus NVFP4 CUDA kernels, managed-memory serving, and FP8-packed KV cache.

Required Runtime

Clone and build the custom ds4 engine:

git clone https://github.com/sleepyeldrazi/ds4-nvfp4-spark
cd ds4-nvfp4-spark
make

Inference (DGX Spark / NVIDIA GB10)

# Managed memory required — saves ~10 GiB RAM, costs 1-3 t/s
DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda -m DeepSeek-V4-Flash-REAP-K150-hybrid.gguf --ctx 131072

# With FP8-packed KV cache
DS4_KV_TURBO=1 DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda \
  -m DeepSeek-V4-Flash-REAP-K150-hybrid.gguf --ctx 262144

API Server

DS4_CUDA_MANAGED_MODEL=1 ./ds4-server --cuda -m DeepSeek-V4-Flash-REAP-K150-hybrid.gguf \
  --host 0.0.0.0 --port 17777 --ctx 131072

Memory Budget (DGX Spark, 128 GB unified memory)

K150 needs managed memory (DS4_CUDA_MANAGED_MODEL=1) to avoid duplication. This saves ~10 GiB of RAM at the cost of 1-3 t/s lower throughput. 1M context requires FP8-packed KV (DS4_KV_TURBO=1).

How It Was Built

1. 1.Source: deepseek-ai/DeepSeek-V4-Flash official MXFP4/MXFP8 mixed-precision checkpoint
2. 2.REAP plan: 8,000 prompts → ds4 imatrix on DGX Spark → activation_energy_sum2 per-expert scores → top-150 per layer (layers 3–42), layers 0–2 preserved at 256
3. 3.NVFP4 repack: gate+up expert e2m1 nibbles copied losslessly from MXFP4; e8m0 per-32 scales converted to e4m3 per-16 + fp32 scale2; down experts kept as Q2K
4. 4.Copy policy: attention, shared experts, output head → Q8_0; embeddings → F16; norms/HC → original precision
5. 5.Emission: single-pass GGUF generation via deepseek4-quantize (NVFP4 emission, REAP pruning, type-change tracking)

No fine-tuning. Purely structural expert removal + lossless NVFP4 repack. Weights are unmodified — a subset of the original MXFP4 experts.

Variants

Acknowledgments

• —DeepSeek — DeepSeek V4 Flash base model
• —antirez — ds4 engine and GGUF quants
• —Cerebras Research — REAP expert pruning
• —NVIDIA — DGX Spark (GB10) hardware
• —eouya2 — reap-for-ds4 tooling