logic65/Qwen3.6-Whittle-25B-A3B

Qwen3.6-Whittle-25B-A3B

Qwen3.6-35B-A3B with 30% of its routed experts removed (256 → 180 per layer) and the damage healed by self-distillation from the unpruned weights. Same architecture (qwen3_5_moe), same tokenizer, same 8-routed + 1-shared experts active per token. Runs on stock transformers and stock llama.cpp, no patches.

How it was made

1. 1.Score every expert on 1M calibration tokens (encyclopaedia, textbooks, maths, code, chat) with a gate-free mean-squared-activation-norm criterion (the task-agnostic winner in the June-2026 one-shot MoE pruning study).
2. 2.Prune the 76 lowest-scoring experts per layer; router rows sliced to match. Kept experts carried 82% of routed traffic on average (71% in the worst layer).
3. 3.Heal for 900 steps × 2048 tokens (≈1.8M tokens) against the unpruned model as teacher — the same weights with the mask off, so no second model and no drift: loss = 2·KL(teacher‖student) + 1·CE. Trained: routers, shared experts, rank-8 LoRA on the kept experts (merged into the exported weights). Optimiser: Muon on 2D matrices, AdamW on the rest. Data: 50% raw corpus windows, 50% template-faithful instruction rows.
4. 4.Export kept experts from the original bf16 shards + LoRA deltas (no quantisation error baked in).

Run it

# transformers ≥ 5.16
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("logic65/Qwen3.6-Whittle-25B-A3B", dtype="bfloat16", device_map="auto")
# llama.cpp (stock): GGUF Q4_K_M in this repo
llama-server -m Qwen3.6-Whittle-25B-A3B-Q4_K_M.gguf -ngl 99 -c 8192 --jinja

Recommended sampling as the parent: temperature 0.7, topp 0.8, topk 20, repeat_penalty 1.05. Thinking mode works as in the parent.

Why is GSM8K higher than the parent?

Eight more correct answers out of 200; sampling noise at this size is about ±2 points, so treat it as "at least parity". Because the heal is also an SFT pass. Half of the healing batches were template-faithful instruction rows, and those include OpenR1-Math reasoning traces, so for 900 steps the model was distilled from its unpruned self and trained on worked maths answers in exactly the no-think, step-by-step format GSM8K is scored in. The parent never had that pass. It is a real gain on this task, not a general one: on raw corpus text the pruned model still sits 0.09 nats of cross-entropy above the parent (1.863 vs 1.771), which is the honest cost of removing 30% of the experts. Expect the same pattern elsewhere: strong on instruction-style tasks close to the healing data, slightly weaker on long-tail knowledge.

What to expect

Twelve heal steps in were enough for "hello" → "Hello! How can I help you today?", one-sentence physics, 17+25=42, and two-turn name recall. The pruned model scores lower CE than the parent on chat-formatted text (calibration and healing both contain chat data) and slightly higher on raw corpus text — the honest damage number is the corpus one. Expect small regressions on long-tail knowledge relative to the 35B; experts that fired rarely on the calibration mix are the ones removed.

next/ — experimental Qwen3.8-Next-format build (work in progress)

next/Qwen3.6-Whittle-25B-A3B-next-EXPERIMENTAL-Q8_0.gguf is the same pruned model converted to the `qwen4exp` architecture: GDN output gate retrained from silu to sigmoid (progressive, back to front, self-distilled), identity hyper-connections, an inert sparse-attention indexer, no n-gram memory yet. It loads and runs on unmodified upstream llama.cpp as qwen4exp. GSM8K on it: 86.5% through llama.cpp (Q80), 82.0% through transformers (bf16) — the gate conversion still costs a few points versus the 92.5% of the silu model above; conversation quality is unchanged. This file is the base the hyper-connections and the n-gram memory are being trained into; treat it as a preview, not a release. `next/sigmoidsettle_step1500.pt` holds the trainable state that produced it.

Support this work

Whittle runs on one hobbyist's grocery budget and rented GPU hours. If this research is useful to you: [ko-fi.com/davida81328](https://ko-fi.com/davida81328) ☕

Authors

David Aylward (logic65) & Claude (Anthropic) — designed, debugged and verified together in one day on a single rented GPU.

Provenance

Parent: Qwen/Qwen3.6-35B-A3B (Apache-2.0). Method references: REAP (Cerebras, arXiv 2510.13999) and "How to Score Experts for One-Shot MoE Expert Pruning" (arXiv 2606.15716). Scripts: logic65/mini-next-a100-kit/colab/ (pruneqwen36.py, healqwen36.py, run_h100.sh). Built on one RTX PRO 6000 Blackwell in about five hours. Part of the Whittle project by logic65.