Daisyamanda/ScienceQA

1---2license: cc-by-sa-4.03annotations_creators:4- expert-generated5- found6language:7- en8language_creators:9- expert-generated10- found11multilinguality:12- monolingual13paperswithcode_id: scienceqa14pretty_name: ScienceQA15size_categories:16- 10K<n<100K17source_datasets:18- original19tags:20- multi-modal-qa21- science22- chemistry23- biology24- physics25- earth-science26- engineering27- geography28- history29- world-history30- civics31- economics32- global-studies33- grammar34- writing35- vocabulary36- natural-science37- language-science38- social-science39task_categories:40- multiple-choice41- question-answering42- other43- visual-question-answering44- text-classification45task_ids:46- multiple-choice-qa47- closed-domain-qa48- open-domain-qa49- visual-question-answering50- multi-class-classification51dataset_info:52  features:53  - name: image54    dtype: image55  - name: question56    dtype: string57  - name: choices58    sequence: string59  - name: answer60    dtype: int861  - name: hint62    dtype: string63  - name: task64    dtype: string65  - name: grade66    dtype: string67  - name: subject68    dtype: string69  - name: topic70    dtype: string71  - name: category72    dtype: string73  - name: skill74    dtype: string75  - name: lecture76    dtype: string77  - name: solution78    dtype: string79  splits:80  - name: train81    num_bytes: 1641690282    num_examples: 1272683  - name: validation84    num_bytes: 540489685    num_examples: 424186  - name: test87    num_bytes: 544167688    num_examples: 424189  download_size: 090  dataset_size: 2726347491---92 93# Dataset Card Creation Guide94 95## Table of Contents96- [Dataset Card Creation Guide](#dataset-card-creation-guide)97  - [Table of Contents](#table-of-contents)98  - [Dataset Description](#dataset-description)99    - [Dataset Summary](#dataset-summary)100    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)101    - [Languages](#languages)102  - [Dataset Structure](#dataset-structure)103    - [Data Instances](#data-instances)104    - [Data Fields](#data-fields)105    - [Data Splits](#data-splits)106  - [Dataset Creation](#dataset-creation)107    - [Curation Rationale](#curation-rationale)108    - [Source Data](#source-data)109      - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization)110      - [Who are the source language producers?](#who-are-the-source-language-producers)111    - [Annotations](#annotations)112      - [Annotation process](#annotation-process)113      - [Who are the annotators?](#who-are-the-annotators)114    - [Personal and Sensitive Information](#personal-and-sensitive-information)115  - [Considerations for Using the Data](#considerations-for-using-the-data)116    - [Social Impact of Dataset](#social-impact-of-dataset)117    - [Discussion of Biases](#discussion-of-biases)118    - [Other Known Limitations](#other-known-limitations)119  - [Additional Information](#additional-information)120    - [Dataset Curators](#dataset-curators)121    - [Licensing Information](#licensing-information)122    - [Citation Information](#citation-information)123    - [Contributions](#contributions)124 125## Dataset Description126 127- **Homepage:** [https://scienceqa.github.io/index.html#home](https://scienceqa.github.io/index.html#home)128- **Repository:** [https://github.com/lupantech/ScienceQA](https://github.com/lupantech/ScienceQA)129- **Paper:** [https://arxiv.org/abs/2209.09513](https://arxiv.org/abs/2209.09513)130- **Leaderboard:** [https://paperswithcode.com/dataset/scienceqa](https://paperswithcode.com/dataset/scienceqa)131- **Point of Contact:** [Pan Lu](https://lupantech.github.io/) or file an issue on [Github](https://github.com/lupantech/ScienceQA/issues)132 133### Dataset Summary134 135Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering136 137### Supported Tasks and Leaderboards138 139Multi-modal Multiple Choice140 141### Languages142 143English144 145## Dataset Structure146 147### Data Instances148 149Explore more samples [here](https://scienceqa.github.io/explore.html).150 151``` json152{'image': Image,153 'question': 'Which of these states is farthest north?',154 'choices': ['West Virginia', 'Louisiana', 'Arizona', 'Oklahoma'],155 'answer': 0,156 'hint': '',157 'task': 'closed choice',158 'grade': 'grade2',159 'subject': 'social science',160 'topic': 'geography',161 'category': 'Geography',162 'skill': 'Read a map: cardinal directions',163 'lecture': 'Maps have four cardinal directions, or main directions. Those directions are north, south, east, and west.\nA compass rose is a set of arrows that point to the cardinal directions. A compass rose usually shows only the first letter of each cardinal direction.\nThe north arrow points to the North Pole. On most maps, north is at the top of the map.',164 'solution': 'To find the answer, look at the compass rose. Look at which way the north arrow is pointing. West Virginia is farthest north.'}165```166 167Some records might be missing any or all of image, lecture, solution. 168 169### Data Fields170 171- `image` : Contextual image172- `question` : Prompt relating to the `lecture`173- `choices` : Multiple choice answer with 1 correct to the `question`174- `answer` : Index of choices corresponding to the correct answer175- `hint` : Hint to help answer the `question`176- `task` : Task description177- `grade` : Grade level from K-12178- `subject` : High level 179- `topic` : natural-sciences, social-science, or language-science180- `category` : A subcategory of `topic`181- `skill` : A description of the task required182- `lecture` : A relevant lecture that a `question` is generated from183- `solution` : Instructions on how to solve the `question`184 185 186Note that the descriptions can be initialized with the **Show Markdown Data Fields** output of the [Datasets Tagging app](https://huggingface.co/spaces/huggingface/datasets-tagging), you will then only need to refine the generated descriptions.187 188### Data Splits189- name: train190    - num_bytes: 16416902191    - num_examples: 12726192- name: validation193    - num_bytes: 5404896194    - num_examples: 4241195- name: test196    - num_bytes: 5441676197    - num_examples: 4241198 199## Dataset Creation200 201### Curation Rationale202 203When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT). This process is normally a black box in the case of deep learning models like large-scale language models. Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system. However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity. To this end, we present Science Question Answering (ScienceQA).204 205### Source Data206 207ScienceQA is collected from elementary and high school science curricula.208 209#### Initial Data Collection and Normalization210 211See Below212 213#### Who are the source language producers?214 215See Below216 217### Annotations218 219Questions in the ScienceQA dataset are sourced from open resources managed by IXL Learning,220an online learning platform curated by experts in the field of K-12 education. The dataset includes221problems that align with California Common Core Content Standards. To construct ScienceQA, we222downloaded the original science problems and then extracted individual components (e.g. questions,223hints, images, options, answers, lectures, and solutions) from them based on heuristic rules.224We manually removed invalid questions, such as questions that have only one choice, questions that225contain faulty data, and questions that are duplicated, to comply with fair use and transformative226use of the law. If there were multiple correct answers that applied, we kept only one correct answer.227Also, we shuffled the answer options of each question to ensure the choices do not follow any228specific pattern. To make the dataset easy to use, we then used semi-automated scripts to reformat229the lectures and solutions. Therefore, special structures in the texts, such as tables and lists, are230easily distinguishable from simple text passages. Similar to ImageNet, ReClor, and PMR datasets,231ScienceQA is available for non-commercial research purposes only and the copyright belongs to232the original authors. To ensure data quality, we developed a data exploration tool to review examples233in the collected dataset, and incorrect annotations were further manually revised by experts. The tool234can be accessed at https://scienceqa.github.io/explore.html.235 236#### Annotation process237 238See above239 240#### Who are the annotators?241 242See above243 244### Personal and Sensitive Information245 246[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)247 248## Considerations for Using the Data249 250### Social Impact of Dataset251 252[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)253 254### Discussion of Biases255 256[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)257 258### Other Known Limitations259 260[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)261 262## Additional Information263 264### Dataset Curators265 266- Pan Lu1,3267- Swaroop Mishra2,3268- Tony Xia1269- Liang Qiu1270- Kai-Wei Chang1271- Song-Chun Zhu1272- Oyvind Tafjord3273- Peter Clark3274- Ashwin Kalyan3275 276From: 2771. University of California, Los Angeles    2782. Arizona State University    2793. Allen Institute for AI   280 281 282 283### Licensing Information284 285[Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)286](https://creativecommons.org/licenses/by-nc-sa/4.0/)287 288### Citation Information289 290Provide the [BibTex](http://www.bibtex.org/)-formatted reference for the dataset. For example:291```292@inproceedings{lu2022learn,293    title={Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering},294    author={Lu, Pan and Mishra, Swaroop and Xia, Tony and Qiu, Liang and Chang, Kai-Wei and Zhu, Song-Chun and Tafjord, Oyvind and Clark, Peter and Ashwin Kalyan},295    booktitle={The 36th Conference on Neural Information Processing Systems (NeurIPS)},296    year={2022}297}298```299### Contributions300 301Thanks to [Derek Thomas](https://huggingface.co/derek-thomas) [@datavistics](https://github.com/datavistics) for adding this dataset.