Modelpublic

steadfastgaze/DeepSeek-V4-Flash-IQ2_XXS-MoEspresso

sourceHugging Facemitupdated 2mo agoView on Hugging Face

0likes18downloads

Model Card

DeepSeek V4 Flash IQ2_XXS, MoEspresso package

This repository holds a MoEspresso package, and MoEspresso is the only engine that serves it: https://github.com/steadfastgaze/MoEspresso.

llama.cpp, transformers, and mlx-lm will not load these files.

Install and serving instructions live in the MoEspresso repository and are deliberately kept out of this card.

Original model: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash

Quality

check	scope	result
Official next-token agreement	Five fixed prompts and 17 greedy next-token decisions, compared with captures from the DeepSeek-V4-Flash API	`17/17` on one tested MLX build; `16/17` on another. The sole difference was lowercase versus uppercase `C`; the reference token was the runner-up locally.
Official continuation loss (lower is better)	100 prompts, 2,290 reference tokens	Average target-token NLL: `0.3765–0.3783`; first-token agreement: `66/100`.
Long-context recall	16 fixed facts in a 30,474-token prompt	`16/16` exact match.

The expert-weight source was compared on the same 100-prompt check:

expert weights	average NLL	first-token agreement
Re-encoded from the bf16 release using the same codec assignments	`0.3915`	`53/100`
Byte-copied from antirez's GGUF	`0.3765`	`66/100`

Speed

Measured on an Apple M3 Max with 128 GB of unified memory - MoEspresso engine.

metric	value
decode	26.06 and 26.24 tok/s, two independent runs on a 3844-token anchor, greedy
time to first token	14.667 s median on the same anchor, about 262 t/s of prefill
longest tested context	113,855 tokens grown in one session; the live cache holds 4.47 GB at that depth

Recipe

The quantization scheme is a faithful reproduction of the IQ2_XXS recipe antirez designed for his ds4 engine, recorded as such in the package's own recipe report. The inputs:

input	source
bf16 safetensors	deepseek-ai/DeepSeek-V4-Flash
recipe GGUF	antirez IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix
imatrix	antirez chat-v2 routed-moe 1p5m, sha256 `02a7c78c2987…`

The routed experts ship byte for byte from antirez's GGUF: gate and up projections in iq2_xxs (86 tensors), down projections in q2_k (43). The 345 dense targets are encoded fresh from the bf16 release into q8_0, following the recipe's assignments: all five attention projection classes, the shared experts, and the output head. The DSv4-specific auxiliary blocks (indexer, compressor, HC) are stored in MoEspresso's affine and mxfp8 formats, and norms pass through in f32.

One clarification the recipe report makes explicit: the imatrix file plays no role in quantization here. The expert bytes are copied and the q80 dense encoding takes no importance weighting. Its only use is ranking experts by observed routing frequency to build `experthotlist.json`, the cold-start prewarm list.

DeepSeek_V4.pdf is the base model's paper and ships because the manifest declares it. package_manifest.json records the sha256 of every file; after downloading, moespresso-verify checks the package against it in one command.

Files

Forty-seven safetensors shards (80.0 GiB, 1328 tensors) plus the sidecars the engine reads: package_manifest.json, package_plan.json, expert_hotlist.json, source_inventory.json, and the recipe report. Tokenizer files come from the original release.

Credits

This quantization is derived from antirez's quantization work for his ds4 engine. The base model is DeepSeek V4 Flash by DeepSeek. Everything in the chain is MIT: the original release, antirez's GGUFs, and this package.