setar007/DeepSeek-V4-Flash-Q8xQ5-GGUF

DeepSeek V4 Flash — Q8xQ5 GGUF

Mixed-precision quantization: Q8_0 (attention, shared expert) + Q5_K (routed experts). Quality equivalent to standard Q5_K_M.

Features

• —Mixed-precision: attention/shared expert/router in Q80, experts in Q5K
• —Per-layer MoE offload: --moe-hot-count auto keeps only needed experts in RAM
• —Typical: 9.7 GB for hot experts (vs 37 GB uniform)
• —Automatically adapts to your workload
• —Full 1M context with MLA compressed KV cache (~7 GB)

Requirements

Stock llama.cpp cannot load this model. You need our custom build:

git clone https://github.com/setar/llama.cpp.git
cd llama.cpp
git checkout feat/moe-expert-persistence
cmake -S . -B build \
  -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON \
  -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=OFF -DGGML_CCACHE=OFF
cmake --build build --config Release --clean-first -j"$(sysctl -n hw.ncpu)"

# For CUDA (NVIDIA):
#   -DGGML_METAL=OFF -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native
#   Replace -j"$(sysctl -n hw.ncpu)" with -j$(nproc)

How to run

./build/bin/llama-server \
  --model DeepSeek-V4-Flash-Instruct-Q8xQ5.gguf-00001-of-00011.gguf \
  --host 0.0.0.0 --port 8082 \
  --ctx-size 1048576 \
  --flash-attn on \
  --batch-size 2048 --ubatch-size 1024 \
  -t 20 --n-gpu-layers all \
  --mlock \
  --moe-hot-count auto \
  --jinja \
  --chat-template-file deepseek-ai-DeepSeek-V4.jinja

Model is split into 11 parts — llama.cpp auto-detects all, just point --model to part 1.

Per-layer MoE offload (--moe-hot-count auto)

Instead of keeping the same number of experts hot in every layer, auto mode tracks which experts are actually used and calculates per-layer optimal counts from accumulated statistics. The data persists across restarts in ~/.llama/expert_<model>.bin.

vs uniform hot-count

Per-layer distribution (from ~466k expert activations)

H0:     4   H1:     4   H2:   128     (hash layers)
S25:  128   S29:     7   S32:    20   (mid scored)
S37:  128   S38:    83   S39:    82   (late scored)
Others:    4                           (deterministic)

Performance (M3 Ultra)

Quantization details

tensor types:
  f32:   535 tensors  (norms, biases, embeddings)
  q8_0:  661 tensors  (attention, shared expert, router)
  q5_K:  129 tensors  (expert FFN: gate/up/down)
  i32:     3 tensors  (expert mapping)
BPW: ~5.2

Files

Links

• —Original model: deepseek-ai/DeepSeek-V4-Flash
• —Custom llama.cpp build: setar/llama.cpp branch feat/moe-expert-persistence
• —Per-layer MoE offload PR: #22694 — closed without review, AI co-author detected
• —llama.cpp upstream: ggml-org/llama.cpp

Shout-out to the llama.cpp maintainers — your automated PR triage is impressively fast, even if it didn't catch that the code was reviewed, tested, and benchmarked before submission. 🤖 → ❤️