leeaandrob/mirror-nvidia__OpenMathInstruct-2

1---2language:3- en4license: cc-by-4.05size_categories:6- 10M<n<100M7task_categories:8- question-answering9- text-generation10pretty_name: OpenMathInstruct-211dataset_info:12  features:13  - name: problem14    dtype: string15  - name: generated_solution16    dtype: string17  - name: expected_answer18    dtype: string19  - name: problem_source20    dtype: string21  splits:22  - name: train_1M23    num_bytes: 135038300324    num_examples: 100000025  - name: train_2M26    num_bytes: 276000967527    num_examples: 200000028  - name: train_5M29    num_bytes: 654649615730    num_examples: 500000031  - name: train32    num_bytes: 1555841297633    num_examples: 1397279134  download_size: 2020892985335  dataset_size: 2621530181136tags:37- math38- nvidia39configs:40- config_name: default41  data_files:42  - split: train43    path: data/train-*44  - split: train_1M45    path: data/train_1M-*46  - split: train_2M47    path: data/train_2M-*48  - split: train_5M49    path: data/train_5M-*50---51 52 53# OpenMathInstruct-254 55OpenMathInstruct-2 is a math instruction tuning dataset with 14M problem-solution pairs 56generated using the [Llama3.1-405B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct) model.57 58The training set problems of [GSM8K](https://github.com/openai/grade-school-math)59and [MATH](https://github.com/hendrycks/math) are used for constructing the dataset in the following ways: 60- *Solution augmentation*: Generating chain-of-thought solutions for training set problems in GSM8K and MATH. 61- *Problem-Solution augmentation*: Generating new problems, followed by solutions for these new problems.      62 63<p>64 <img src="SFT Data Diagram 1.jpg" width="75%" title="Composition of OpenMathInstruct-2">65</p>66 67OpenMathInstruct-2 dataset contains the following fields:68 69- **problem**: Original problem from either the GSM8K or MATH training set or augmented problem from these training sets.70- **generated_solution**: Synthetically generated solution.71- **expected_answer**: For problems in the training set, it is the ground-truth answer provided in the datasets. **For augmented problems, it is the majority-voting answer.**72- **problem_source**: Whether the problem is taken directly from GSM8K or MATH or is an augmented version derived from either dataset.73 74  75<p>76 <img src="scaling_plot.jpg" width="40%" title="Scaling Curve">77</p>78 79We also release the 1M, 2M, and 5M, *fair-downsampled* versions of the entire training set corresponding to points in the above scaling plot. 80These splits are referred to as **train_1M**, **train_2M**, and **train_5M**. 81To use these subsets, just specify one of these subsets as split while downloading the data:82```python83from datasets import load_dataset84 85# Download only the 1M training split86dataset = load_dataset('nvidia/OpenMathInstruct-2', split='train_1M', streaming=True)87```  88 89To download the entire training set and to convert it into the jsonl format, use the following code snippet.90This might take 20-30 minutes (or more depending on your network connection) and will use ~20Gb of RAM.91```python92import json93 94from datasets import load_dataset95from tqdm import tqdm96 97dataset = load_dataset('nvidia/OpenMathInstruct-2', split='train')98 99print("Converting dataset to jsonl format")100output_file = "openmathinstruct2.jsonl"101with open(output_file, 'w', encoding='utf-8') as f:102    for item in tqdm(dataset):103        f.write(json.dumps(item, ensure_ascii=False) + '\n')104 105print(f"Conversion complete. Output saved as {output_file}")106```107 108Apart from the dataset, we also release the [contamination explorer](https://huggingface.co/spaces/nvidia/OpenMathInstruct-2-explorer) for looking at problems 109in the OpenMathInstruct-2 dataset that are similar to the [GSM8K](https://huggingface.co/datasets/openai/gsm8k), [MATH](https://github.com/hendrycks/math), 110[AMC 2023](https://github.com/QwenLM/Qwen2.5-Math/tree/main/evaluation/data/amc23), [AIME 2024](https://artofproblemsolving.com/wiki/index.php/2024_AIME_I), 111and [Omni-MATH](https://huggingface.co/datasets/KbsdJames/Omni-MATH) test set problems. 112 113See our [paper](https://arxiv.org/abs/2410.01560) to learn more details!114 115 116### Note 117The released dataset doesn't filter out extremely long questions. After the dataset release, we found that 564 questions (roughly 0.1%) were longer than 1024 Llama tokens. 118We experimented with removing these questions and didn't see a performance drop (in fact, we observed a minor bump). Dropping these questions, helps with memory as well. 119So we would recommend, filtering out extremely long questions. We have updated the data preparation commands in our [Github documentation](https://nvidia.github.io/NeMo-Skills/openmathinstruct2/dataset/#converting-to-sft-format).   120 121 122## OpenMath2 models123 124To demonstrate the quality of this dataset, we release a series of OpenMath2 models trained on this data.125 126| Model | GSM8K | MATH | AMC 2023 | AIME 2024 | Omni-MATH |127|:---|:---:|:---:|:---:|:---:|:---:|128| Llama3.1-8B-Instruct | 84.5 | 51.9 | 9/40 | 2/30 | 12.7 |129| OpenMath2-Llama3.1-8B ([nemo](https://huggingface.co/nvidia/OpenMath2-Llama3.1-8B-nemo) \| [HF](https://huggingface.co/nvidia/OpenMath2-Llama3.1-8B)) | 91.7 | 67.8 | 16/40 | 3/30 | 22.0 |130| + majority@256 | 94.1 | 76.1 | 23/40 | 3/30 | 24.6 |131| Llama3.1-70B-Instruct | 95.8 | 67.9 | 19/40 | 6/30 | 19.0 |132| OpenMath2-Llama3.1-70B ([nemo](https://huggingface.co/nvidia/OpenMath2-Llama3.1-70B-nemo) \| [HF](https://huggingface.co/nvidia/OpenMath2-Llama3.1-70B)) | 94.9 | 71.9 | 20/40 | 4/30 | 23.1 |133| + majority@256 | 96.0 | 79.6 | 24/40 | 6/30 | 27.6 |134 135The pipeline we used to produce the data and models is fully open-sourced!136 137- [Code](https://github.com/NVIDIA/NeMo-Skills)138- [Models](https://huggingface.co/collections/nvidia/openmath-2-66fb142317d86400783d2c7b)139- [Dataset](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2)140 141## Reproducing our results142 143We provide [all instructions](https://nvidia.github.io/NeMo-Skills/openmathinstruct2/)144to fully reproduce our results, including data generation.145 146## Citation147 148If you find our work useful, please consider citing us!149 150```bibtex151@article{toshniwal2024openmath2,152  title   = {OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data},153  author  = {Shubham Toshniwal and Wei Du and Ivan Moshkov and  Branislav Kisacanin and Alexan Ayrapetyan and Igor Gitman},154  year    = {2024},155  journal = {arXiv preprint arXiv:2410.01560}156}157```