lightonai/Qwen3-8B-ZH-Swap

Qwen3-8B-ZH-Swap

Qwen3-8B-ZH-Swap is a Layer Swap model built on top of `lightonai/Qwen3-8B-ZH`: the middle transformer layers (L13–L20) of the English specialist `lightonai/Qwen3-8B-EN` have been transplanted into the Chinese native specialist. The resulting model reasons natively in Chinese while inheriting the stronger reasoning core of the English specialist.

It is released alongside the paper **Rethinking the Multilingual Reasoning Gap with Layer Swap**.

Model details

• —Base model: Qwen/Qwen3-8B-Base
• —Construction: Training-free Layer Swap — layers L13–L20 of Qwen3-8B-EN transplanted into Qwen3-8B-ZH
• —Language: Chinese (CoT and answer)
• —Context length: 32,768 tokens
• —Dataset (underlying specialists): `lightonai/Dolci-Think-SFT-32B-Multilingual`

[!NOTE] The model was trained on data derived from allenai/Dolci-Think-SFT-32B, released under the ODC-BY-1.0 license.

Related models

This model is part of a Chinese specialist trio designed to study the native reasoning gap:

Evaluation

All scores are mean accuracy (%) on the Chinese version of each benchmark, with sample standard deviation across runs. AIME 24/25 is averaged over 30 runs; the others over 10 runs, using the recommended generation parameters.

Benchmarks used:

• —`lightonai/gpqa_diamond_multilingual`
• —`lightonai/aime24_multilingual`
• —`lightonai/aime25_multilingual`
• —`lightonai/HumanEvalPlus_multilingual`
• —`lightonai/mgsm-rev2`
• —`CohereLabs/Global-MMLU-Lite`

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "lightonai/Qwen3-8B-ZH-Swap"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "计算：24 × 17 = ?"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)

outputs = model.generate(inputs, max_new_tokens=32768, temperature=1.0, top_p=0.95, top_k=20)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

Recommended sampling: temperature=1.0, top_p=0.95, top_k=20, min_p=0.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{lasbordes2026rethinking,
  title        = {Rethinking the Multilingual Reasoning Gap with Layer Swap},
  author       = {Lasbordes, Maxence and Chatelain, Amélie and Seddah, Djamé},
  year         = {2026},
  eprint       = {2605.26735},
  archivePrefix= {arXiv},
  primaryClass = {cs.CL}
}