sepsy070716/Qwen3.5-4B-A3B-Multilingual-Distilled-v1
Qwen3.5-4B-A3B-Multilingual-Distilled-v1
Experimental multilingual layerwise-distilled checkpoint derived from sepsy070716/Qwen3.5-4B-A3B-Upcycled with Qwen/Qwen3.5-4B as the dense teacher.
Research checkpoint. All 32 MoE blocks received local dense-FFN distillation, but the complete language model has not yet received joint end-to-end recovery training. Local reconstruction gains are not equivalent to downstream benchmark gains.
Architecture
Multilingual distillation
Training used a balanced ODC-By-1.0 FineWeb/FineWeb2 sample in Korean, English, Mandarin Chinese, Japanese, Spanish, and German. Every layer received 24 local updates (four documents per language). Evaluation used a disjoint split with two documents per language and layer.
- Mean held-out dense-FFN relative-MSE improvement: 6.57%
- Layers improved: 32/32
- Every language improved on every layer: 192/192 comparisons
Language means: de 6.83%, en 5.12%, es 7.15%, ja 7.47%, ko 7.42%, zh 5.60%.
End-to-end language-model check
On four disjoint 128-token documents per language, mean causal-LM loss improved from 10.378 for the untrained upcycle to 9.162 for this checkpoint, an 11.72% relative reduction. All six languages improved. The dense teacher scored about 2.94, which quantifies the large remaining recovery gap.
A two-prompt greedy generation smoke test loaded and executed successfully, but the next four tokens were whitespace-dominated in both Korean and English. Therefore this checkpoint is not yet suitable for chat, completion, benchmark, or production claims. Joint end-to-end recovery is required.
Loading
from transformers import AutoTokenizer, Qwen3_5MoeForConditionalGeneration
model_id = "sepsy070716/Qwen3.5-4B-A3B-Multilingual-Distilled-v1"
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(model_id, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)See distillation_manifest.json and distillation_summary.json for complete per-layer and per-language measurements.