ducanhdinh/Qwen1.5-MoE-A2.7B-Finetuning

Qwen1.5-MoE-A2.7B-Finetuning

Day la LoRA adapter finetune tu `Qwen/Qwen1.5-MoE-A2.7B`, mot mo hinh Mixture-of-Experts.

Cau hinh LoRA

• —Layer duoc finetune: [8, 16) trong tong so 24 layer (tuong ung khoang 1L/3 -> 2L/3).
• —Module duoc gan LoRA: attention, router, experts trong khoang layer tren, moi thanh phan mot rank rieng qua rank_pattern cua PEFT:
• —attention: r = 16
• —router: r = 4
• —experts: r = 16
• —alpha = 32, dropout = 0.05

Loss

Loss MoE tieu chuan:

L_total = L_LM + lb_loss_coef * L_LB

• —L_LM: cross-entropy chuan tren token tiep theo.
• —L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.
• —lb_loss_coef = None
• —num_experts = 60, top_k = 4

Du lieu

Alignment data duoc chon qua --alignment_data (flores ntrex ted), gom sample tu cac file: flores.json, ntrex.json, ted.json (moi field ngon ngu trong 1 record duoc coi la 1 sample), shuffle va sort theo do dai token truoc khi gom batch.

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step), diagnostics/loss_curve.png (bieu do tho LLM / LLB / LTotal theo tung step) va `diagnostics/losscurve_smoothed.png (cung 3 duong loss nhung da lay trung binh cong moi 50` step lien tiep — de doc xu huong hon vi bieu do tho rat messy o cap do tung step).