xik94/DeepSeek-V4-Flash-162B-REAP-GGUF

DeepSeek-V4-Flash-162B REAP - GGUF Quantizations (Multi-GPU)

Multi-GPU GGUF quantizations of DeepSeek-V4-Flash-162B-REAP for llama.cpp.

Tested on 2x RTX 4090 (modificated to 48GB VRAM each, 96GB total) with CUDA sm_89 (Ada Lovelace).

These files REQUIRE a patched llama.cpp - stock llama.cpp crashes after 2-3 prompts due to non-unique expert IDs in REAP's tid2eid routing tensors (issues #24591, #25598, labeled wontfix).

TLDR: Download the GGUF + patched files from patches/, build llama.cpp with CUDA, run with --tensor-split 1,1 --split-mode layer. See instructions below.

Files

License

This model inherits the MIT License from the original DeepSeek-V4-Flash-162B by 0xSero. The llama.cpp patches are also MIT-licensed (llama.cpp is MIT).

Source

• —Original model: 0xSero/DeepSeek-V4-Flash-162B (BF16 safetensors)
• —Inference engine: llama.cpp (MIT License)
• —Architecture: DeepSeek-V4 with REAP pruning (284B -> 162B)
• —MoE: 144 experts, MXFP4 expert weights, MHC (Manifold Hyper-Connections)

CRITICAL: You NEED the patched llama.cpp

The REAP model has non-unique expert IDs in tid2eid routing tensors. Stock llama.cpp crashes after 2-3 prompts with:

CUDA error: illegal memory access

This is a known issue labeled wontfix by llama.cpp maintainers because REAP's routing is non-standard.

Patch 1: Fix duplicate expert ID crash in mmid.cu

File: ggml/src/ggml-cuda/mmid.cu

Problem: The mm_ids_helper kernel uses warp_reduce_any which only checks if ANY thread in the warp matched. When the same expert appears twice for one token (as REAP does), multiple threads try to write to the same memory slot, causing a race condition and illegal memory access.

Fix: Replace warp_reduce_any with a warp-level prefix sum (inclusive scan). This:

1. 1.Counts the total number of matching threads per token (not just 0/1)
2. 2.Gives each matching thread a unique write offset in the store array
3. 3.The kernel correctly handles N copies of the same expert for one token

The patch modifies two code paths:

• —Generic path (n_expert_used_template == 0): Uses full-warp prefix sum
• —Optimized path (templated expert count): Uses prefix sum within each neu_padded group

Full patched file is in patches/mmid.cu in this repo. The unified diff is in patches/mmid.cu.patch.

Patch 2: Exclude i32 routing tensors from quantization

File: src/llama-quant.cpp

Problem: When requantizing from one GGUF format to another, llama.cpp tries to dequantize all tensors. The tid2eid tensors are i32 type and cannot be dequantized, causing:

llama_model_quantize: failed to quantize: cannot dequantize/convert tensor type i32

Fix: Add one line to tensor_allows_quantization():

// do not quantize i32 routing tensors (DeepSeek V4 tid2eid)
quantize &= name.find("tid2eid") == std::string::npos;

Full diff in patches/llama-quant.cpp.patch.

Build Instructions (Multi-GPU)

# 1. Clone llama.cpp (latest version)
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

# 2. Copy patched files over the originals
# Download mmid.cu and llama-quant.cpp from the patches/ folder in this repo
cp mmid.cu ggml/src/ggml-cuda/mmid.cu
cp llama-quant.cpp src/llama-quant.cpp

# 3. Build with CUDA
# Adjust CUDA_ARCHITECTURES for your GPU:
#   89 = RTX 4090 (Ada)
#   86 = RTX 3090 (Ampere)
#   90 = H100 (Hopper)
mkdir build && cd build
cmake .. -DGGML_CUDA=ON \
         -DCMAKE_CUDA_ARCHITECTURES=89 \
         -DCMAKE_BUILD_TYPE=Release \
         -DLLAMA_BUILD_TESTS=OFF \
         -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build . -j$(nproc) --target llama-server llama-quantize

Running (Multi-GPU)

Q2_K with 200K context (RECOMMENDED - best speed + max context)

./build/bin/llama-server \
  --model DSV4-Flash-162B-REAP-Q2_K.gguf \
  --ctx-size 204800 \
  --n-gpu-layers 99 \
  --tensor-split 1,1 \
  --split-mode layer \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --port 8032 --host 0.0.0.0 \
  -t 8 --parallel 1 -fa on

Q3KM with 128K context

./build/bin/llama-server \
  --model DSV4-Flash-162B-REAP-Q3_K_M.gguf \
  --ctx-size 131072 \
  --n-gpu-layers 99 \
  --tensor-split 1,1 \
  --split-mode layer \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --port 8032 --host 0.0.0.0 \
  -t 8 --parallel 1 -fa on

Benchmark Results (2x RTX 4090 48GB, 96GB VRAM, Multi-GPU)

Hardware Requirements

• —Minimum: 2x GPU with 48GB VRAM each (96GB total) - e.g. 2x RTX 4090 modificated to 48GB VRAM each
• —Architecture: sm_89 (Ada Lovelace) or newer
• —System RAM: 112GB+ recommended
• —Disk: 56-73GB per quantization file
• —Multi-GPU: Required - uses --tensor-split 1,1 --split-mode layer to distribute across 2 GPUs

Technical Details

Why is DSv4 slower than other MoE models?

DSv4 uses a novel architecture with multiple complex attention mechanisms per layer:

• —CSA (Compressed Sparse Attention) with Lightning Indexer
• —HCA (Heavily Compressed Attention)
• —MHC (Manifold Hyper-Connections) with Sinkhorn normalization
• —MLA (Multi-head Latent Attention) for KV cache compression

This results in ~5x more operations per layer compared to standard transformer MoE models like MiniMax-M2.7 (which runs at 95 tok/s on the same hardware).

Acknowledgments

• —Original model: 0xSero/DeepSeek-V4-Flash-162B
• —Inference engine: llama.cpp
• —REAP prunes the full 284B DeepSeek-V4 to 162B while retaining quality