meta-math/MetaMath-Mistral-7B

1---2license: apache-2.03datasets:4- meta-math/MetaMathQA5---6see our paper in https://arxiv.org/abs/2309.122847 8View the project page:9https://meta-math.github.io/10 11## Note12 13All MetaMathQA data are augmented from the training sets of GSM8K and MATH. 14<span style="color:red"><b>None of the augmented data is from the testing set.</b></span>15 16You can check the `original_question` in `meta-math/MetaMathQA`, each item is from the GSM8K or MATH train set.17 18## Model Details19 20MetaMath-Mistral-7B is fully fine-tuned on the MetaMathQA datasets and based on the powerful Mistral-7B model. It is glad to see using MetaMathQA datasets and change the base model from llama-2-7B to Mistral-7b can boost the GSM8K performance from 66.5 to **77.7**.21 22To fine-tune Mistral-7B, I would suggest using a smaller learning rate (usually 1/5 to 1/10 of the lr for LlaMa-2-7B) and staying other training args unchanged.23More training details and scripts can be seen at https://github.com/meta-math/MetaMath24 25## Installation26 27```28pip install transformers==4.35.029pip install torch==2.0.130pip install sentencepiece==0.1.9931pip install tokenizers==0.13.332pip install accelerate==0.21.033pip install bitsandbytes==0.40.034pip install vllm35pip install fraction36pip install protobuf37```38 39## Model Usage40 41prompting template:42 43'''44 45"Below is an instruction that describes a task. "46"Write a response that appropriately completes the request.\n\n"47"### Instruction:\n{instruction}\n\n### Response: Let's think step by step."48 49'''50 51where you need to use your query question to replace the {instruction} 52 53There is another interesting repo about Arithmo-Mistral-7B in https://huggingface.co/akjindal53244/Arithmo-Mistral-7B, where they combine our MetaMathQA dataset and MathInstruct datasets to train a powerful model. Thanks agian for their contributions.54We would also try to train the combination of **MetaMathQA** and **MathInstruct** datasets, and also open all the results and training details.55 56## Experiments57 58| Model               | GSM8k Pass@1 | MATH Pass@1 |59|---------------------|--------------|-------------|60| MPT-7B              | 6.8          | 3.0         |61| Falcon-7B           | 6.8          | 2.3         |62| LLaMA-1-7B          | 11.0         | 2.9         |63| LLaMA-2-7B          | 14.6         | 2.5         |64| MPT-30B             | 15.2         | 3.1         |65| LLaMA-1-13B         | 17.8         | 3.9         |66| GPT-Neo-2.7B        | 19.5         | --          |67| Falcon-40B          | 19.6         | 2.5         |68| Baichuan-chat-13B   | 23.9         | --          |69| Vicuna-v1.3-13B     | 27.6         | --          |70| LLaMA-2-13B         | 28.7         | 3.9         |71| InternLM-7B         | 31.2         | --          |72| ChatGLM-2-6B        | 32.4         | --          |73| GPT-J-6B            | 34.9         | --          |74| LLaMA-1-33B         | 35.6         | 3.9         |75| LLaMA-2-34B         | 42.2         | 6.24        |76| RFT-7B              | 50.3         | --          |77| LLaMA-1-65B         | 50.9         | 10.6        |78| Qwen-7B             | 51.6         | --          |79| WizardMath-7B       | 54.9         | 10.7        |80| LLaMA-2-70B         | 56.8         | 13.5        |81| WizardMath-13B      | 63.9         | 14.0        |82| MAmmoTH-7B (COT)    | 50.5         | 10.4        |83| MAmmoTH-7B (POT+COT)| 53.6         | 31.5        |84| Arithmo-Mistral-7B  | 74.7         | 25.3        |85| MetaMath-7B         | 66.5         | 19.8        |86| MetaMath-13B        | 72.3         | 22.4        |87| 🔥 **MetaMath-Mistral-7B** | **77.7**     | **28.2**        |88 89## Citation90 91```bibtex92@article{yu2023metamath,93  title={MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models},94  author={Yu, Longhui and Jiang, Weisen and Shi, Han and Yu, Jincheng and Liu, Zhengying and Zhang, Yu and Kwok, James T and Li, Zhenguo and Weller, Adrian and Liu, Weiyang},95  journal={arXiv preprint arXiv:2309.12284},96  year={2023}97}98```99 100```bibtex101@article{jiang2023mistral,102  title={Mistral 7B},103  author={Jiang, Albert Q and Sablayrolles, Alexandre and Mensch, Arthur and Bamford, Chris and Chaplot, Devendra Singh and Casas, Diego de las and Bressand, Florian and Lengyel, Gianna and Lample, Guillaume and Saulnier, Lucile and others},104  journal={arXiv preprint arXiv:2310.06825},105  year={2023}106}107```