erokhins/Qwen3.6-27B-FFN-pruned-25

Qwen3.6-27B-FFN-pruned-25

A structurally pruned Qwen3.6-27B: 25% of every FFN layer's intermediate channels are physically removed, shrinking intermediate_size from 17408 → 13056 (exactly 0.75×). Weights are pure Qwen3.6 — no merging — but which channels were removed was decided by comparing Qwen3.6 against Qwen3.8.

Method: drift-guided structured pruning

Qwen3.8-27B is a continued-training descendant of Qwen3.6-27B: every FFN channel sits at the same index in both, and no channel was replaced (minimum same-index cosine across the whole model is ~0.52). That makes per-channel cross-version drift measurable and usable as a pruning signal.

For each FFN channel c in layer L:

score(L,c) = min( cos(W_gate³·⁶[c,:], W_gate³·⁸[c,:]),
                  cos(W_up³·⁶[c,:],   W_up³·⁸[c,:]),
                  cos(W_down³·⁶[:,c], W_down³·⁸[:,c]) )

Worst-of-three: a channel only counts as stable if its gate row, up row and down column all survived continued training.

Selection (`p2030r25`)

1. 1.Protect the most stable channels, with a depth ramp: top 20% of layer 0 rising linearly to top 30% of layer 63 (25% of all channels protected). The ramp follows the measured stability gradient — mean drift cosine rises from ~0.72 in early layers to ~0.91 in the last, so late layers hold more genuinely converged channels.
2. 2.Cut 25% of each layer at random from the unprotected pool. Removing a channel means deleting its gate row, up row and down column together, which is exact: the SwiGLU intermediate dimension has no cross-channel interaction, so channels can be permuted and truncated without changing the function of what remains.

Why random inside the pool, rather than "cut the most-drifted"? Controlled experiments found drift to be a weak selection signal — at equal volume, cutting the most-drifted channels performed no better than random. But it is a strong protection signal: cutting the most stable channels instead caused severe fluency collapse at just 30%. Stability marks converged, load-bearing machinery. So the protection set does the work and random picking fills the quota.

The MTP head (mtp.layers.0) carries its own 17408-wide SwiGLU, so it is pruned the same way and to the same width (its channels scored a mean drift cosine of 0.850, comparable to the main layers; protection set to 30%, matching the ramp's end since the head sits after the last layer). mtp.fc, the MTP norms and its attention are untouched.

Usage

Standard transformers / mlx-vlm loading — the config declares the reduced intermediate_size, so no custom code is needed.

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "erokhins/Qwen3.6-27B-FFN-pruned-25", dtype="bfloat16", device_map="auto")
processor = AutoProcessor.from_pretrained("erokhins/Qwen3.6-27B-FFN-pruned-25")

Run it with thinking disabled (enable_thinking: false in the chat template). In non-thinking mode it answers cleanly and concisely. With thinking enabled it tends to over-deliberate — note that unpruned Qwen3.6 is also verbose in thinking mode, so pruning amplifies an existing trait rather than creating it.

Limitations

• —Evaluated only by smoke testing, not by perplexity or benchmark suites. Treat quality claims as provisional.
• —Pruning damage in this family shows up first in the direct-answer pathway and as occasional dropped short tokens, not as gibberish — watch for terse or empty responses rather than incoherence.
• —Aggressive pruning of the GDN mixers was tried separately and damaged real usage even at 12.5%; this model leaves them alone. FFN-only is the safe cut here.
• —Not instruction-tuned or fine-tuned after pruning. A least-squares reconstruction of W_down on calibration activations would likely recover some quality and is the obvious next step.

Reproducing

Scripts, masks and a full experiment write-up (including the pruning-cliff study: the unprotected cliff sits near 50% of FFN channels, ~57% with flat protection, ~60% with this depth-adaptive scheme) are in the project repository.

# 1. per-channel drift scores from the two parent checkpoints
uv run compare_36_38.py
# 2. the p2030r25 mask
uv run make_hybrid_mask.py --scheme protect --frac 25 \
    --floor-ramp 20 30 --pick random --uniform-count --seed 25 \
    --out null_mask_p2030r25.npy
# 4. extend the mask with a row for the MTP head's FFN, then build & truncate
uv run add_mtp_to_mask.py null_mask_p2030r25.npy null_mask_p2030r25_mtp.npy 30 25
COEFF1=1.0 uv run build_null_merge.py zeros-27B null_mask_p2030r25_mtp.npy
uv run truncate_model.py zeros-27B Qwen3.6-27B-FFN-pruned-25 \
    null_mask_p2030r25_mtp.npy 13056

Attribution

Derived from Qwen/Qwen3.6-27B (Apache-2.0); channel selection additionally used Qwen/Qwen3.8-27B for comparison only — none of its weights are present in this model. Released under Apache-2.0, as the base model.