miweru/Inkling-Small-REAP-137B-A12B-de-GGUF

Inkling-Small-REAP-137B-A12B-de (GGUF)

A REAP expert-pruned variant of thinkingmachines/Inkling-Small (276B-A12B → 137B total / 12B active), provided as GGUF quantizations.

Variants in this repo

The importance matrix was computed on a German-heavy DE/EN mix matching the pruning calibration (German multi-register human text, German model outputs, English code and math), so the quantization error budget follows the same language priorities as the expert pruning. Both variants are split into ≤45 GB parts; keep the parts of one variant in the same directory and load the first part.

The idea: produce a sparse model in the 100–140B class with a moderate quantization level (Q4KM, no aggressive sub-4-bit tricks), so that the quality loss budget is spent once on expert pruning and once on a well-understood quantization — not twice on extreme compression. We plan to spot-benchmark it against dense/sparse models of similar deployed size (e.g. Qwen3.5-122B-class models).

⚠️ Language disclaimer: the expert-pruning calibration was optimized for German and English (German-heavy multi-register calibration mix). Losses in other languages are presumably significantly higher — experts that were mostly used by other languages were preferentially removed. If you need multilingual coverage beyond DE/EN, use the original Inkling-Small.

What was done

• —Method: REAP (Router-weighted Expert Activation Pruning, Lasby et al., ICLR 2026) — one-shot, no retraining. Per MoE layer, the 128 of 256 routed experts with the lowest router-weighted activation saliency were removed; router rows / e-score bias sliced accordingly. Shared experts, attention, dense layers, embeddings and MTP weights are untouched.
• —Calibration: ~4M tokens, 56 % German (human-written text across 10 registers — news, legal, parliamentary, scientific, encyclopedic, spoken, easy language, essays, social, narrative — plus German instruction/chat data and German outputs of frontier models), 44 % English code / reasoning / tool-calling / chat.
• —Quantization: llama.cpp Q4KM from BF16.

Calibration only collects routing statistics; no calibration text enters the weights.

Internal retention probes (not benchmarks)

Teacher-forced streaming probes against the BF16 base model (ΔNLL in nats, top-1 agreement with base). Spot checks, not standardized benchmarks:

For comparison: an English-only-calibrated pruning of the same ratio degraded German prose by +1.67 nats. Proper benchmark results (vs. Qwen3.5-122B-class models) are planned.

Usage

Requires a llama.cpp build with Inkling support — currently the `add-inkling` PR #25731 until it is merged:

git clone -b add-inkling https://github.com/danielhanchen/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
./build/bin/llama-cli -m Inkling-Small-REAP-137B-A12B-de-Q4_K_M-00001-of-00002.gguf \
  -p "Erkläre den Unterschied zwischen Bundesrat und Bundestag." -n 300

The split GGUF parts must sit in the same directory; llama.cpp loads them automatically from the first part.

Limitations

• —DE/EN optimized — other languages likely degrade well beyond typical REAP losses (see disclaimer above).
• —Code/tool-calling gives up a little quality relative to an English/code-only calibration (that trade-off was chosen deliberately).
• —Text-only: vision/audio towers of the multimodal original are not part of this GGUF.
• —No MTP speculative decoding in this GGUF (llama.cpp Inkling MTP support pending). The underlying pruned safetensors checkpoint retains the full MTP stack.
• —Pruned models remain probabilistic compressions of the original — verify before production use.

About the base model: Inkling-Small

⚠️ Carry-over disclaimer: Everything in this section describes the original thinkingmachines/Inkling-Small (276B-A12B, BF16, multimodal). This repo contains a 50 %-expert-pruned, Q4_K_M-quantized, text-only derivative — the properties and scores below do not all carry over. Treat every number as an upper bound, not a claim about this model. Multimodal inputs are absent here entirely, and multilingual ability outside German/English was deliberately traded away during pruning calibration.

Key facts (original)

Selected original benchmark scores

Scores of the unpruned BF16 original, from the official model card (where it is compared against Qwen3.5-397B-A17B, MiMo V2.5, Minimax M2.7 and closed models — see there for the full 10-model table). None of these numbers have been re-measured for this pruned+quantized variant.

Multimodal scores (MMMU Pro, Charxiv, MMAU, VoiceBench …) are omitted: the vision/audio towers are not part of this GGUF.

Deployment of the original

The original model has day-0 recipes for SGLang, vLLM, TokenSpeed, Unsloth and Hugging Face transformers (links in the original card), including MTP speculative decoding. This GGUF instead targets llama.cpp (see Usage above); our pruned safetensors checkpoint (TML layout, MTP intact, vLLM/SGLang-compatible) is not published here — open a discussion if you need it.

Safety, bias and limitations (inherited)

The original card documents safety training, known biases, risks and legal terms — all of that applies here at best unchanged and plausibly degraded: expert pruning is a lossy intervention whose effect on safety behavior and refusal calibration has not been separately evaluated. Review the original card's Safety / Bias / Legal sections before production use.

Provenance

• —Base: thinkingmachines/Inkling-Small (Apache-2.0)
• —Pruning: REAP streaming pipeline (calibration ran block-wise on a single RTX 5090 workstation without ever loading the 532 GB model into memory)
• —Per-layer pruned expert lists: see reap_pruning.json in this repo

@inproceedings{lasby2026reap,
  title={{REAP} the Experts: Why Pruning Prevails for One-Shot MoE compression},
  author={Lasby, Lazarevich, Sinnadurai, Lie, Ioannou, Thangarasa},
  booktitle={ICLR}, year={2026}
}

Deutsch (Kurzfassung)

REAP-geprunte Variante von Inkling-Small: pro Layer wurden die 128 von 256 Experten mit der geringsten router-gewichteten Aktivierungs-Salienz entfernt (one-shot, kein Training), anschließend Q4KM-Quantisierung. Die Kalibrierung war bewusst deutschlastig (56 %, menschliche Texte über zehn Register plus deutsche Instruktionsdaten): deutsche Prosa verliert nur +0,39 nats gegenüber dem Original (statt +1,67 bei englischer Kalibrierung), deutscher Sachtext liegt auf Originalniveau. In anderen Sprachen als Deutsch und Englisch sind die Verluste mutmaßlich deutlich höher. Geplant sind Stichproben-Benchmarks gegen Modelle der Qwen3.5-122B-Klasse.