sleepyeldrazi/DeepSeek-v4-Flash-REAP-K180-NVFP4

DeepSeek V4 Flash — REAP K180 Mixed NVFP4 (NVFP4 + Q2_K + Q8)

REAP-pruned DeepSeek V4 Flash at K180 (180 routed experts per MoE layer, 30% pruning), with a hybrid mixed-precision quantization targeting NVIDIA DGX Spark (GB10 Blackwell, sm_121). Supports full 1M token context on a single 128 GB DGX Spark via managed memory + FP8-packed KV cache.

At a Glance

Mixed Quantization Structure

This model uses a hybrid quant strategy optimized for DGX Spark's bandwidth profile (273 GB/s LPDDR5X, ~97 GB/s managed-memory serving path):

Why this mix?

The bottleneck in MoE decode is gate+up expert projection (~2/3 of expert bytes). Standard GGUF quants use IQ2XXS here. NVFP4 reads more bytes (4.50 vs 2.06 bpw) but offers substantially better precision at the same bit budget, with dequant bandwidth at ~140 GB/s on GB10. In practice, NVFP4 decode is slightly slower than IQ2XXS due to the higher byte volume, but the precision gain is well worth the small throughput cost. Down experts stay Q2K (already saturating at ~160 GB/s). Attention and shared experts stay Q80 for quality.

NVFP4: lossless MXFP4→NVFP4 conversion

The HF source checkpoint stores experts in MXFP4 (e2m1 nibbles + e8m0 per-32 block scale). NVFP4 is e2m1 + e4m3 per-16 + per-expert fp32 scale_2. The e2m1 nibbles are identical — the conversion is a scale-only transform. No weight requantization, no quality loss from the quantization step. The NVFP4 weights are bit-identical to the MXFP4 originals; only the scale format changes.

Domain Split (Calibration)

8,000 prompts · 5.0M tokens · calibration on DGX Spark (NVIDIA GB10) at 4,096 token context. REAP activation_energy_sum2 score metric.

How to Run

⚠️ Standard llama.cpp / GGUF runtimes will NOT load this model. It uses a custom NVFP4 multi-tensor GGUF convention (.nvfp4_weight + .nvfp4_scale_2) plus NVFP4 CUDA kernels, managed-memory serving, and FP8-packed KV cache.

Required Runtime

Clone and build the custom ds4 engine:

git clone https://github.com/sleepyeldrazi/ds4-nvfp4-spark
cd ds4-nvfp4-spark
make

The managed-memory path (required for K180)

The DGX Spark (GB10) has hardware ATS — the GPU reads CPU-allocated memory directly through the host page tables, coherently, no copy. The ds4-nvfp4-spark runtime uses cudaMallocManaged + cudaMemAdvise(SetReadMostly + SetPreferredLocation=device) + cudaMemPrefetchAsync to achieve ~97 GB/s single-residency serving, avoiding the 2× duplication of the original ds4's cudaMemcpy span cache. This saves ~10 GiB of RAM but costs 1-3 t/s depending on context length — the fault-and-migrate overhead scales with working set churn.

Without `DS4_CUDA_MANAGED_MODEL=1`, K180 will OOM.

Inference (DGX Spark / NVIDIA GB10)

# Short context — managed memory required
DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda \
  -m DeepSeek-V4-Flash-REAP-K180-hybrid.gguf --ctx 32768

# 256K context — managed memory + FP8 KV
DS4_KV_TURBO=1 DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda \
  -m DeepSeek-V4-Flash-REAP-K180-hybrid.gguf --ctx 262144

# Full 1M context — managed memory + FP8 KV
DS4_KV_TURBO=1 DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda \
  -m DeepSeek-V4-Flash-REAP-K180-hybrid.gguf --ctx 1048576

API Server

DS4_KV_TURBO=1 DS4_CUDA_MANAGED_MODEL=1 ./ds4-server --cuda \
  -m DeepSeek-V4-Flash-REAP-K180-hybrid.gguf \
  --host 0.0.0.0 --port 17777 --ctx 1048576

Memory Budget (DGX Spark, 128 GB unified memory)

Resident figures include the ~7 GiB fixed UVM page-tracking overhead. The FP8 KV cache at 584 bytes/row (vs 2048 for FP32) saves ~7.2 GiB at 1M context.

At 1M context with FP8 KV, K180 peaks at 116 GiB — under the 128 GB physical limit with ~12 GiB headroom for the OS. The UVM overhead (7 GiB) is a fixed cost, not reclaimable by Linux THP (UVM memory isn't THP-eligible).

Confirming single residency

# After loading, verify no duplication:
nvidia-smi  # "Memory-Usage: Not Supported" is expected on iGPU
# Check process RSS — should be ~model size + KV + overhead, not 2×:
grep VmRSS /proc/$(pgrep ds4)/status

How It Was Built

1. 1.Source: deepseek-ai/DeepSeek-V4-Flash official MXFP4/MXFP8 mixed-precision checkpoint
2. 2.REAP plan: 8,000 prompts → ds4 imatrix on DGX Spark → activation_energy_sum2 per-expert scores → top-180 per layer (layers 3–42), layers 0–2 preserved at 256
3. 3.NVFP4 repack: gate+up expert e2m1 nibbles copied losslessly from MXFP4; e8m0 per-32 scales converted to e4m3 per-16 + fp32 scale2; down experts kept as Q2K
4. 4.Copy policy: attention, shared experts, output head → Q8_0; embeddings → F16; norms/HC → original precision
5. 5.Emission: single-pass GGUF generation via deepseek4-quantize (NVFP4 emission, REAP pruning, type-change tracking)

No fine-tuning. Purely structural expert removal + lossless NVFP4 repack. Weights are unmodified — a subset of the original MXFP4 experts.

Why REAP + Hybrid NVFP4?

Standard GGUF quants (IQ2XXS, Q2K, Q4K) are compact but sacrifice precision at low bit widths. NVFP4 trades some speed for substantially better quality: it reads more bytes per weight (4.50 vs 2.06 bpw) but preserves the lossless MXFP4→NVFP4 mapping, meaning no requantization error on expert weights. The result is a model that prioritizes precision over raw throughput — slightly slower decode than an all-IQ2XXS quant, but with dramatically better weight fidelity on the expert path.

Combined with REAP (30% expert pruning), managed memory, and FP8-packed KV:

• —Fits K180 on a single 128 GB DGX Spark
• —Serves full 1M token context
• —Preserves attention quality (Q8_0)
• —Has no quantization quality loss on experts (lossless MXFP4→NVFP4)

The GB10 has no TMEM, no WGMMA, no tcgen05 — it's consumer Blackwell (sm121), not datacenter Blackwell. The NVFP4 `_dp4a kernel (~140 GB/s) is the pragmatic v1; a native mma.mxf4` kernel (projected ~186–210 GB/s) is the next step.

For complete bandwidth analysis and kernel benchmarks, see the accompanying cuda_debug/ANALYSIS.md in the runtime repo.

Variants

Acknowledgments

• —DeepSeek — DeepSeek V4 Flash base model
• —antirez — ds4 engine and GGUF quants
• —Cerebras Research — REAP expert pruning
• —NVIDIA — DGX Spark (GB10) hardware
• —eouya2 — reap-for-ds4 tooling