barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-GGUF

Qwen3.6-29B-REAP-Opus-Reasoning-Distill-GGUF

GGUF quantizations of a REAP-pruned (205/256 experts) Qwen3.6-35B-A3B MoE, merged with an Opus-reasoning-distilled LoRA adapter. This is the plain merge — no Multi-Token Prediction (MTP) head.

A version with an MTP head grafted on (for self-speculative decoding) is available at barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-GGUF. See the benchmarks below to decide which one fits your hardware.

Converted with llama.cpp's convert_hf_to_gguf.py.

Highlights

• —The reasoning LoRA beats the unpruned base model on ARC-Challenge (0.616 vs 0.532) — a net skill gain from distillation, not just recovered pruning loss. See Quality benchmarks.
• —Identical speed to the MTP release with speculative decoding off (219.06 vs 219.34 tok/s, full offload) — no downside to picking this file unless you specifically want the MTP speedup. See Benchmarks.

Files

All quants ≤Q4KM were calibrated with an importance matrix (imatrix) built from 512 samples of barozp/opus-reasoning-distill-train. K-quants (Q*_K*) favor broad compatibility and fast CPU inference; IQ-quants (IQ*) require the imatrix and give better quality per bit at ≤4-bit, at some CPU-inference speed cost.

Why choose this over the MTP version?

The MTP-grafted version carries an extra decoder layer used for self-speculative decoding. When that feature is left enabled on hardware that's memory-bandwidth-constrained (e.g. a small/weaker GPU with heavy CPU offload), the draft+verify overhead can compete with an already-scarce resource and result in slower generation than this plain model. This release removes that footgun entirely — no toggle to remember, always the same speed as the MTP file with speculative decoding off.

Benchmarks

Measured with llama-cli (Q4_K_M, flash attention on, greedy decoding, 5 runs per config, mean ± std) on an NVIDIA RTX PRO 6000 Blackwell Server Edition (97 GB VRAM). Full methodology and the MTP comparison are on the MTP-GGUF model card.

This matches the MTP-GGUF file with speculative decoding disabled (--spec-type none) within measurement noise — confirming the MTP head, when unused, carries no VRAM/compute penalty. If your setup benefits from speculative decoding (compute-bound, strong GPU, full offload), the MTP release may be faster; see its model card for numbers (+39–67% observed in our tests).

Quality benchmarks

Measured with lm-evaluation-harness (HF backend, bfloat16, chat template disabled — see note) on the underlying safetensors checkpoint, against the same base chain: unpruned Qwen3.6-35B-A3B → REAP 205/256 pruning only (no LoRA, RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2) → this checkpoint.

Key finding: the reasoning LoRA doesn't just recover REAP's pruning loss on ARC-Challenge — it pushes the score above the unpruned 256-expert base model (0.616 vs 0.532), a genuine reasoning-skill transfer from the Opus chain-of-thought training data (ARC-Challenge appears in neither REAP's calibration mixture nor the LoRA's training data). MMLU sees a smaller but real gain (+1.3pp retained vs. the LoRA-less pruned checkpoint); wikitext perplexity is unaffected; HellaSwag is flat within measurement noise.

Note: chat template was tested and found to badly corrupt loglikelihood-based multiple-choice scoring for this model family (MMLU dropped from 0.85 to 0.38 on the base model with it on) — all numbers above are with it off, applied consistently across every checkpoint. HellaSwag/ARC-Challenge are 0-shot, also applied consistently. Full methodology on the [safetensors model card](https://huggingface.co/barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill#quality-benchmarks).

Credits

• —Base architecture: Qwen3.6-35B-A3B (MoE), pruned via REAP (Router-weighted Expert Activation Pruning) to 205/256 experts.
• —Reasoning distillation: LoRA fine-tune on Opus-generated reasoning traces (barozp/opus-reasoning-distill-train).