sepsy070716/Qwen3.5-4B-A3B-Upcycled

Qwen3.5-4B-A3B-Upcycled

Experimental sparse-MoE initialization derived from Qwen/Qwen3.5-4B.

Research checkpoint, not a trained release. The dense FFNs were compressed and the experts have not undergone continued pretraining or distillation. Expect quality loss relative to the base model. Do not treat benchmark results from the base model as results for this checkpoint.

Architecture

The original 9,216-wide dense FFN is reduced to a 1,536-wide shared expert and eight 704-wide routed experts with top-2 routing. Neurons are ranked per layer by the product of their gate, up, and down projection norms. The strongest shared and routed slices are retained. Routed experts start identically so the untrained router does not make the initial function nondeterministic.

Intended use

This checkpoint is intended as an initialization for router warm-up, knowledge distillation from Qwen/Qwen3.5-4B, and continued pretraining. It is not recommended for production or user-facing inference before recovery training and evaluation.

Load with a Transformers release that provides Qwen3_5MoeForConditionalGeneration:

from transformers import Qwen3_5MoeForConditionalGeneration, AutoProcessor

model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    "sepsy070716/Qwen3.5-4B-A3B-Upcycled",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("sepsy070716/Qwen3.5-4B-A3B-Upcycled")

See conversion_manifest.json and neuron_selection.json for reproducibility.

Router warm-up v1

A router-only MPS warm-up artifact is published under research/router-warmup-v1/. It updates 655,360 router parameters and leaves all attention, expert, embedding, and vision weights untouched.

On 32 held-out FineWeb2 Korean documents (8,109 valid tokens), normalized routing entropy improved from 0.99599 to 0.99748, while the maximum/minimum expert usage ratio improved from 1.528 to 1.390. This adapter balances routing but does not recover the quality lost by dense-FFN compression; expert distillation is still required.

Layerwise distillation pilots

Accepted layer adapters for depths 0, 16, and 31 are published under research/layer-distillation-pilots/. Each was trained for 10 local updates with the v1 router frozen. On eight held-out Korean documents, dense-FFN relative MSE improved by 4.07%, 8.24%, and 8.33% respectively, with improvement on all 24 document/layer comparisons.

These local reconstruction results are proof of direction, not an end-to-end model benchmark. The root checkpoint has not been modified by the pilot adapters.