openai/gsm8k

1---2annotations_creators:3- crowdsourced4language_creators:5- crowdsourced6language:7- en8license:9- mit10multilinguality:11- monolingual12size_categories:13- 1K<n<10K14source_datasets:15- original16task_categories:17- text-generation18task_ids: []19paperswithcode_id: gsm8k20pretty_name: Grade School Math 8K21tags:22- math-word-problems23dataset_info:24- config_name: main25  features:26  - name: question27    dtype: string28  - name: answer29    dtype: string30  splits:31  - name: train32    num_bytes: 396320233    num_examples: 747334  - name: test35    num_bytes: 71373236    num_examples: 131937  download_size: 272563338  dataset_size: 467693439- config_name: socratic40  features:41  - name: question42    dtype: string43  - name: answer44    dtype: string45  splits:46  - name: train47    num_bytes: 519810848    num_examples: 747349  - name: test50    num_bytes: 93685951    num_examples: 131952  download_size: 316425453  dataset_size: 613496754configs:55- config_name: main56  data_files:57  - split: train58    path: main/train-*59  - split: test60    path: main/test-*61- config_name: socratic62  data_files:63  - split: train64    path: socratic/train-*65  - split: test66    path: socratic/test-*67---68 69# Dataset Card for GSM8K70 71## Table of Contents72- [Dataset Description](#dataset-description)73  - [Dataset Summary](#dataset-summary)74  - [Supported Tasks](#supported-tasks-and-leaderboards)75  - [Languages](#languages)76- [Dataset Structure](#dataset-structure)77  - [Data Instances](#data-instances)78  - [Data Fields](#data-instances)79  - [Data Splits](#data-instances)80- [Dataset Creation](#dataset-creation)81  - [Curation Rationale](#curation-rationale)82  - [Source Data](#source-data)83  - [Annotations](#annotations)84  - [Personal and Sensitive Information](#personal-and-sensitive-information)85- [Considerations for Using the Data](#considerations-for-using-the-data)86  - [Social Impact of Dataset](#social-impact-of-dataset)87  - [Discussion of Biases](#discussion-of-biases)88  - [Other Known Limitations](#other-known-limitations)89- [Additional Information](#additional-information)90  - [Dataset Curators](#dataset-curators)91  - [Licensing Information](#licensing-information)92  - [Citation Information](#citation-information)93 94## Dataset Description95 96- **Homepage:** https://openai.com/blog/grade-school-math/97- **Repository:** https://github.com/openai/grade-school-math98- **Paper:** https://arxiv.org/abs/2110.1416899- **Leaderboard:** [Needs More Information]100- **Point of Contact:** [Needs More Information]101 102### Dataset Summary103 104GSM8K (Grade School Math 8K) is a dataset of 8.5K high quality linguistically diverse grade school math word problems. The dataset was created to support the task of question answering on basic mathematical problems that require multi-step reasoning.105- These problems take between 2 and 8 steps to solve.106- Solutions primarily involve performing a sequence of elementary calculations using basic arithmetic operations (+ − ×÷) to reach the final answer.107- A bright middle school student should be able to solve every problem: from the paper, "Problems require no concepts beyond the level of early Algebra, and the vast majority of problems can be solved without explicitly defining a variable."108- Solutions are provided in natural language, as opposed to pure math expressions. From the paper: "We believe this is the most generally useful data format, and we expect it to shed light on the properties of large language models’ internal monologues""109 110### Supported Tasks and Leaderboards111 112This dataset is generally used to test logic and math in language modelling.113It has been used for many benchmarks, including the [LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).114 115### Languages116 117The text in the dataset is in English. The associated BCP-47 code is `en`.118 119## Dataset Structure120 121### Data Instances122 123For the `main` configuration, each instance contains a string for the grade-school level math question and a string for the corresponding answer with multiple steps of reasoning and calculator annotations (explained [here](https://github.com/openai/grade-school-math#calculation-annotations)).124 125 126```python127{128    'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?',129    'answer': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72',130}131```132 133For the `socratic` configuration, each instance contains a string for a grade-school level math question, a string for the corresponding answer with multiple steps of reasoning, calculator annotations (explained [here](https://github.com/openai/grade-school-math#calculation-annotations)), and *Socratic sub-questions*.134 135```python136{137    'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?',138    'answer': 'How many clips did Natalia sell in May? ** Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nHow many clips did Natalia sell altogether in April and May? ** Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72',139}140```141 142### Data Fields143 144The data fields are the same among `main` and `socratic` configurations and their individual splits.145 146- question: The question string to a grade school math problem.147 148- answer: The full solution string to the `question`. It contains multiple steps of reasoning with calculator annotations and the final numeric solution.149 150### Data Splits151 152| name   |train|validation|153|--------|----:|---------:|154|main    | 7473|      1319|155|socratic| 7473|      1319|156 157## Dataset Creation158 159### Curation Rationale160 161[Needs More Information]162 163### Source Data164 165#### Initial Data Collection and Normalization166 167From the paper, appendix A:168 169> We initially collected a starting set of a thousand problems and natural language solutions by hiring freelance contractors on Upwork (upwork.com). We then worked with Surge AI (surgehq.ai), an NLP data labeling platform, to scale up our data collection. After collecting the full dataset, we asked workers to re-solve all problems, with no workers re-solving problems they originally wrote. We checked whether their final answers agreed with the original solutions, and any problems that produced disagreements were either repaired or discarded. We then performed another round of agreement checks on a smaller subset of problems, finding that 1.7% of problems still produce disagreements among contractors. We estimate this to be the fraction of problems that contain breaking errors or ambiguities. It is possible that a larger percentage of problems contain subtle errors.170 171#### Who are the source language producers?172 173[Needs More Information]174 175### Annotations176 177#### Annotation process178 179[Needs More Information]180 181#### Who are the annotators?182 183Surge AI (surgehq.ai)184 185### Personal and Sensitive Information186 187[Needs More Information]188 189## Considerations for Using the Data190 191### Social Impact of Dataset192 193[Needs More Information]194 195### Discussion of Biases196 197[Needs More Information]198 199### Other Known Limitations200 201[Needs More Information]202 203## Additional Information204 205### Dataset Curators206 207[Needs More Information]208 209### Licensing Information210 211The GSM8K dataset is licensed under the [MIT License](https://opensource.org/licenses/MIT).212 213### Citation Information214 215```bibtex216@article{cobbe2021gsm8k,217  title={Training Verifiers to Solve Math Word Problems},218  author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},219  journal={arXiv preprint arXiv:2110.14168},220  year={2021}221}222```223 224### Contributions225 226Thanks to [@jon-tow](https://github.com/jon-tow) for adding this dataset.