sepsy070716/Qwen3.5-4B-A3B-Student-v2

Qwen3.5-4B-A3B-Student-v2

A text-only sparse-MoE release candidate built as a practical local alternative to Qwen3.5-4B. It has 4.0B total parameters and 3.0B active parameters per token. The initialization preserves Qwen3.5-2B behavior, while adding output-neutral trainable capacity for later Qwen3.5-4B distillation.

This is an independently measured research release. It is not an official Qwen model and does not include vision.

Architecture

Each MoE layer initially computes one half of the original Qwen3.5-2B dense MLP through the shared path and one half through the selected routed expert. The two routed experts begin functionally identical. Additional neurons have random gate/up projections and zero down projections, making them output-neutral but trainable. Exact conversion metadata is in conversion_manifest.json.

Evaluation

All reported results were produced locally on an Apple M4 with 32GB unified memory. Raw JSON reports are included in evaluation/.

Chat and sentence generation

The fixed gate contains 60 prompts: 10 each in Korean, English, Chinese, Japanese, Spanish, and German. It covers facts, arithmetic, translation, instruction following, and free-form sentence generation.

Six chemical-formula answers used the correct Unicode spelling H₂O; the scorer normalizes Unicode subscripts before comparison.

Multilingual held-out LM loss

Four held-out FineWeb/FineWeb2 documents per language, 128 tokens per document:

Standard benchmark development subset

EleutherAI lm-evaluation-harness==0.4.12, zero-shot, BF16, first 100 examples per task. These limited results are development indicators, not full-task benchmark claims.

The subset mean is 90.6% of the Qwen3.5-4B teacher mean.

Local service gate

The included FastAPI service completed 20/20 consecutive non-streaming POST /v1/chat/completions requests:

Requests generated up to 16 new tokens. See evaluation/openai_service_20.json.

Transformers usage

Use Transformers 5.13.0 or another version that provides Qwen3_5MoeForCausalLM:

import torch
from transformers import AutoTokenizer, Qwen3_5MoeForCausalLM

model_id = "sepsy070716/Qwen3.5-4B-A3B-Student-v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = Qwen3_5MoeForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
    return_dict=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Reproduction and service

research_code/ contains the converter, multilingual loss comparison, 60-prompt gate and scorer, plus the local OpenAI-compatible service and its 20-request test. The service implements GET /health, GET /v1/models, and non-streaming POST /v1/chat/completions.

Limitations

• —Current quality is inherited primarily from Qwen3.5-2B; the extra capacity has not yet received large-scale continued pretraining or teacher distillation.
• —The 100-example ARC-Easy/HellaSwag figures are small development subsets and have substantial sampling uncertainty. Run the full tasks before making publication or production claims.
• —This model is text-only and cannot replace the original model's vision path.
• —The included server is a single-process local research server. It has no authentication, TLS, streaming, tool calling, or multi-worker support.
• —Apply the same safety, bias, privacy, and factuality evaluation required for any deployment of the upstream Qwen models.

License and attribution

Released under Apache-2.0, following the included upstream license. Derived from Qwen3.5-2B weights and evaluated against Qwen3.5-4B. Qwen model names and trademarks belong to their respective owners.