SoFarSoGoodya/DeepMath-R1-Distill-Qwen-7B

1---2license: mit3library_name: peft4pipeline_tag: text-generation5base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B6tags:7  - lora8  - sft9  - dpo10  - math11  - reasoning12  - llama-factory13  - deepseek-r114  - qwen15datasets:16  - AI-MO/NuminaMath-CoT17language:18  - en19  - zh20---21 22# DeepMath (LoRA SFT + DPO on DeepSeek-R1-Distill-Qwen-7B)23 24> 🇨🇳 [中文简介](#中文) below.25 26**DeepMath** is a math-reasoning fine-tune of [`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B), produced by a **two-stage LoRA pipeline** with **LLaMA-Factory** (template `deepseekr1`):27 28```29DeepSeek-R1-Distill-Qwen-7B ──SFT (LoRA, NuminaMath-CoT)──► merged ──DPO (LoRA, math prefs)──► DeepMath30```31 32This repo contains **LoRA adapters**, not a merged model. It was built in a 5-day sprint on rented consumer GPUs (8×5090 for SFT, 1×5090 for DPO) — the full engineering story is in the [GitHub repo](https://github.com/SoFarSoGoodya/DeepMath).33 34## ⚠️ What's in here & how to use it35 36There are **two adapters**, applied **in sequence**:37 38| Path | Role | Size |39|---|---|---|40| `sft_adapter/` | Stage 1 — SFT LoRA (NuminaMath-CoT) | 78 MB |41| `./` (top level) | Stage 2 — DPO LoRA (math preference pairs), the **final** adapter | 77 MB |42 43The DPO adapter was trained on top of the *SFT-merged* model, so to reconstruct the final model you must apply **SFT first, then DPO**. Use the included `merge.py`:44 45```bash46pip install transformers peft torch accelerate47python merge.py \48  --base deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \49  --sft  ./sft_adapter \50  --dpo  ./ \51  --out  ./DeepMath-merged52```53 54Or load adapters on the fly (see `merge.py` docstring / the GitHub `examples/inference.py`).55 56## Training summary (honest)57 58| Stage | Config | Outcome |59|---|---|---|60| **SFT** | 8×5090, native DDP, LoRA r=8/α=16 (target=all), lr 5e-5, 1 epoch, cutoff 2048, 1,410 steps | loss **0.73 → 0.39** |61| **DPO** | 1×5090, **QLoRA** 4-bit NF4 + double-quant, lr 1e-6, 3 epochs, cutoff 1024, 456 steps | loss **0.6947** (≈ ln 2 baseline), reward acc ~0.5–0.55 |62 63![SFT loss](sft_training_loss.png)64![DPO reward accuracy](dpo_rewards_accuracies.png)65 66## Limitations & intended use67 68- **This is a learning/pipeline-demonstration artifact, not a SOTA model.** SFT converged cleanly; DPO completed without mode collapse but its gain is **modest** (reward accuracy only slightly above the 50% baseline) and **no held-out benchmark was evaluated**.69- Best understood as a reproducible reference for a resource-constrained SFT→DPO pipeline. Don't deploy it expecting strong math-reasoning gains.70- Base model `DeepSeek-R1-Distill-Qwen-7B` inherits its biases/limits; tokenizer derives from Qwen.71 72## Provenance & license73 74- **Base**: [`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B) (MIT).75- **SFT data**: [`AI-MO/NuminaMath-CoT`](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT) (Apache-2.0), cleaned 100k→90,217.76- **DPO data**: 2,418 math preference pairs (distilabel-math-preference style).77- **Adapters + code**: **MIT**. Please retain upstream attributions.78 79**Author**: [@SoFarSoGoodya](https://github.com/SoFarSoGoodya) — full attribution & contact in the [GitHub repo](https://github.com/SoFarSoGoodya/DeepMath).80 81---82 83## 中文84 85**DeepMath** 是基于 `DeepSeek-R1-Distill-Qwen-7B` 的数学推理微调,用 **LLaMA-Factory** 走 **LoRA SFT(NuminaMath)→ 合并 → LoRA DPO(数学偏好对)** 两段式流水线。86 87本仓库是 **LoRA adapter**(非合并模型)。含两个 adapter,需**按顺序**使用:先 `sft_adapter/`(SFT),再顶层目录(DPO,最终)。用自带的 `merge.py` 一条命令重建完整模型(见上方命令)。88 89**如实说明**:这是一个学习/流水线演示作品,**不是 SOTA 模型**。SFT 收敛良好;DPO 完整跑通但提升有限(奖励准确率仅略高于随机),且未做独立评测。权重与代码采用 **MIT**;请保留对上游(DeepSeek / Qwen / NuminaMath)的署名。完整工程复盘见 [GitHub 仓库](https://github.com/SoFarSoGoodya/DeepMath)。90