barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill

Qwen3.6-29B-REAP-Opus-Reasoning-Distill

A REAP-pruned (205/256 experts) Qwen3.6-35B-A3B Mixture-of-Experts model, LoRA fine-tuned on Claude Opus reasoning traces and merged (not a standalone adapter). ~28.24B total parameters, ~3B active per token (routing behavior unchanged from the base MoE).

This is the plain merge — no Multi-Token Prediction (MTP) head. A version with an MTP head grafted on (for self-speculative decoding in llama.cpp) is at barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP. GGUF quantizations of this checkpoint are at barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-GGUF.

Highlights

• —The reasoning LoRA beats the unpruned base model on ARC-Challenge (0.616 vs 0.532) — it doesn't just offset REAP's pruning loss, it's a net skill gain, despite ARC-Challenge appearing in neither REAP's calibration data nor the LoRA's training data.
• —96.9% MMLU retention at 20% expert pruning — comfortably inside the empirically-measured safe zone (95% retention crosses at ~27-28% pruning for this calibration).
• —Full methodology, more pruning ratios, and the perplexity-vs-accuracy divergence finding: see Quality benchmarks below.

Pipeline

1. 1.REAP pruning — 51/256 experts removed from Qwen/Qwen3.6-35B-A3B via REAP (Router-weighted Expert Activation Pruning), criterion = router weight × expert activation norm, calibrated on a code/math/tool-use mixture.
2. 2.LoRA reasoning distillation — rank 64, alpha 64, targeting attention projections (10 full-attention layers), shared-expert projections (all 40 layers), and fused routed-expert tensors (all 40 layers). Trained on barozp/opus-reasoning-distill-train (14,250 Claude Opus reasoning traces), then merged directly into the weights.

Quality benchmarks

Measured with lm-evaluation-harness (HF backend, bfloat16, chat template disabled — see note) on the same base checkpoint chain: unpruned Qwen3.6-35B-A3B → REAP 205/256 pruning only (no LoRA, RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2) → this checkpoint.

Key finding: the reasoning LoRA doesn't just recover REAP's pruning loss on ARC-Challenge — it pushes the score above the unpruned 256-expert base model (0.616 vs 0.532). ARC-Challenge appears in neither REAP's calibration mixture nor the LoRA's training data, so this is a genuine reasoning-skill transfer from the Opus chain-of-thought traces. MMLU sees a smaller but real gain (+1.3pp retained vs. the LoRA-less pruned checkpoint); wikitext perplexity is unchanged (the LoRA targets reasoning ability, not general language-modeling quality); HellaSwag is flat within measurement noise (500-sample eval).

Note on methodology: chat template was tested and found to badly corrupt loglikelihood-based multiple-choice scoring for this model family (e.g. MMLU dropped from 0.85 to 0.38 on the base model with chat template on) — all numbers above are measured with it off, applied consistently across every checkpoint in the comparison. HellaSwag/ARC-Challenge are 0-shot (lower than typical 10-shot/25-shot leaderboard numbers), also applied consistently.

REAP pruning quality curve (context)

For reference, REAP's own quality-vs-ratio curve on this exact calibration setup (MMLU, out-of-calibration axis):

The 95%-retention crossing point is around 27-28% pruning — 20% sits comfortably inside the safe zone with modest headroom.

Credits

• —Base architecture: Qwen3.6-35B-A3B (MoE), pruned via REAP (Router-weighted Expert Activation Pruning) to 205/256 experts.
• —Reasoning distillation: LoRA fine-tune on Opus-generated reasoning traces (barozp/opus-reasoning-distill-train).