OctoThinker/OctoThinker-3B-Short-Base

1---2license: llama3.23datasets:4- OctoThinker/MegaMath-Web-Pro-Max5- LLM360/MegaMath6language:7- en8base_model:9- meta-llama/Llama-3.2-3B10pipeline_tag: text-generation11---12 13# [OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling](https://arxiv.org/abs/2506.20512)14 15 16 17## OctoThinker-3B-Short-Base18 19 20The OctoThinker family is built on carefully studied mid-training insights, starting from the Llama-3 family, to create a reinforcement learning–friendly base language model.21 22### Training Recipe23 24<div style="display: flex; justify-content: left; gap: 20px;">25<img src="https://cdn-uploads.huggingface.co/production/uploads/62cbeb2d72dfd24b86bdf977/2sFzePngjjopTs0SeCS9R.png" alt="Data Pipeline" style="width:90%;">26</div>27 28 29 30### Evaluation Results31 32Note that we adopt the few-shot prompting evaluation for these base language models.33 34 35<div style="display: flex; justify-content: left; gap: 20px;">36<img src="https://cdn-uploads.huggingface.co/production/uploads/62cbeb2d72dfd24b86bdf977/UCZ9MahRYqLY0iKjiWMqS.png" alt="Data Pipeline" style="width:80%;">37 38</div>39 40 41### More about OctoThinker42 43 44<div style="display: flex; justify-content: left; gap: 20px;">45<img src="https://cdn-uploads.huggingface.co/production/uploads/62cbeb2d72dfd24b86bdf977/bn85CEB_DW6azJ7KJp11Q.png" alt="Data Pipeline" style="width:100%;">46</div>47 48 49## Citation50 51Check out our [paper](https://arxiv.org/abs/2506.20512) for more details. If you use our models, datasets or find our work useful, please cite52 53```54@article{wang2025octothinker,55  title={OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling},56  author={Wang, Zengzhi and Zhou, Fan and Li, Xuefeng and Liu, Pengfei},57  year={2025},58  journal={arXiv preprint arXiv:2506.20512},59  note={Preprint}60}61```62