sydclone/qasper

1---2pretty_name: QASPER3annotations_creators:4- expert-generated5language_creators:6- expert-generated7language:8- en9language_bcp47:10- en-US11license:12- cc-by-4.013multilinguality:14- monolingual15size_categories:16- 10K<n<100K17source_datasets:18- extended|s2orc19task_categories:20- question-answering21task_ids:22- closed-domain-qa23paperswithcode_id: qasper24---25 26# Dataset Card for Qasper27 28## Table of Contents29  - [Table of Contents](#table-of-contents)30  - [Dataset Description](#dataset-description)31    - [Dataset Summary](#dataset-summary)32    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)33    - [Languages](#languages)34  - [Dataset Structure](#dataset-structure)35    - [Data Instances](#data-instances)36    - [Data Fields](#data-fields)37    - [Data Splits](#data-splits)38  - [Dataset Creation](#dataset-creation)39    - [Curation Rationale](#curation-rationale)40    - [Source Data](#source-data)41      - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization)42      - [Who are the source language producers?](#who-are-the-source-language-producers)43    - [Annotations](#annotations)44      - [Annotation process](#annotation-process)45      - [Who are the annotators?](#who-are-the-annotators)46    - [Personal and Sensitive Information](#personal-and-sensitive-information)47  - [Considerations for Using the Data](#considerations-for-using-the-data)48    - [Social Impact of Dataset](#social-impact-of-dataset)49    - [Discussion of Biases](#discussion-of-biases)50    - [Other Known Limitations](#other-known-limitations)51  - [Additional Information](#additional-information)52    - [Dataset Curators](#dataset-curators)53    - [Licensing Information](#licensing-information)54    - [Citation Information](#citation-information)55    - [Contributions](#contributions)56 57## Dataset Description58 59- **Homepage:** [https://allenai.org/data/qasper](https://allenai.org/data/qasper)60- **Demo:** [https://qasper-demo.apps.allenai.org/](https://qasper-demo.apps.allenai.org/)61- **Paper:** [https://arxiv.org/abs/2105.03011](https://arxiv.org/abs/2105.03011)62- **Blogpost:** [https://medium.com/ai2-blog/question-answering-on-scientific-research-papers-f6d6da9fd55c](https://medium.com/ai2-blog/question-answering-on-scientific-research-papers-f6d6da9fd55c)63- **Leaderboards:** [https://paperswithcode.com/dataset/qasper](https://paperswithcode.com/dataset/qasper)64 65### Dataset Summary66 67QASPER is a dataset for question answering on scientific research papers. It consists of 5,049 questions over 1,585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting evidence to answers.68 69### Supported Tasks and Leaderboards70 71- `question-answering`: The dataset can be used to train a model for Question Answering. Success on this task is typically measured by achieving a *high* [F1 score](https://huggingface.co/metrics/f1). The [official baseline model](https://github.com/allenai/qasper-led-baseline) currently achieves 33.63 Token F1 score & uses [Longformer](https://huggingface.co/transformers/model_doc/longformer.html). This task has an active leaderboard which can be found [here](https://paperswithcode.com/sota/question-answering-on-qasper)72 73- `evidence-selection`: The dataset can be used to train a model for Evidence Selection. Success on this task is typically measured by achieving a *high* [F1 score](https://huggingface.co/metrics/f1). The [official baseline model](https://github.com/allenai/qasper-led-baseline) currently achieves 39.37 F1 score & uses [Longformer](https://huggingface.co/transformers/model_doc/longformer.html). This task has an active leaderboard which can be found [here](https://paperswithcode.com/sota/evidence-selection-on-qasper)74 75 76### Languages77 78English, as it is used in research papers.79 80## Dataset Structure81 82### Data Instances83 84A typical instance in the dataset:85 86```87{88  'id': "Paper ID (string)",89  'title': "Paper Title",90  'abstract': "paper abstract ...",91  'full_text': {92      'paragraphs':[["section1_paragraph1_text","section1_paragraph2_text",...],["section2_paragraph1_text","section2_paragraph2_text",...]],93      'section_name':["section1_title","section2_title"],...},94  'qas': {95  'answers':[{96      'annotation_id': ["q1_answer1_annotation_id","q1_answer2_annotation_id"]97      'answer': [{98          'unanswerable':False,99          'extractive_spans':["q1_answer1_extractive_span1","q1_answer1_extractive_span2"],100          'yes_no':False,101          'free_form_answer':"q1_answer1",102          'evidence':["q1_answer1_evidence1","q1_answer1_evidence2",..],103          'highlighted_evidence':["q1_answer1_highlighted_evidence1","q1_answer1_highlighted_evidence2",..]104          },105          {106          'unanswerable':False,107          'extractive_spans':["q1_answer2_extractive_span1","q1_answer2_extractive_span2"],108          'yes_no':False,109          'free_form_answer':"q1_answer2",110          'evidence':["q1_answer2_evidence1","q1_answer2_evidence2",..],111          'highlighted_evidence':["q1_answer2_highlighted_evidence1","q1_answer2_highlighted_evidence2",..]112          }],113      'worker_id':["q1_answer1_worker_id","q1_answer2_worker_id"]114      },{...["question2's answers"]..},{...["question3's answers"]..}],115  'question':["question1","question2","question3"...],116  'question_id':["question1_id","question2_id","question3_id"...],117  'question_writer':["question1_writer_id","question2_writer_id","question3_writer_id"...],118  'nlp_background':["question1_writer_nlp_background","question2_writer_nlp_background",...],119  'topic_background':["question1_writer_topic_background","question2_writer_topic_background",...],120  'paper_read': ["question1_writer_paper_read_status","question2_writer_paper_read_status",...],121  'search_query':["question1_search_query","question2_search_query","question3_search_query"...],122  }123}124```125 126### Data Fields127 128The following is an excerpt from the dataset README:129 130Within "qas", some fields should be obvious. Here is some explanation about the others:131 132#### Fields specific to questions:133 134 - "nlp_background" shows the experience the question writer had. The values can be "zero" (no experience), "two" (0 - 2 years of experience), "five" (2 - 5 years of experience), and "infinity" (> 5 years of experience). The field may be empty as well, indicating the writer has chosen not to share this information.135 136 - "topic_background" shows how familiar the question writer was with the topic of the paper. The values are "unfamiliar", "familiar", "research" (meaning that the topic is the research area of the writer), or null.137 138 - "paper_read", when specified shows whether the questionwriter has read the paper.139 140 - "search_query", if not empty, is the query the question writer used to find the abstract of the paper from a large pool of abstracts we made available to them.141 142#### Fields specific to answers143 144Unanswerable answers have "unanswerable" set to true. The remaining answers have exactly one of the following fields being non-empty.145 146 - "extractive_spans" are spans in the paper which serve as the answer.147 - "free_form_answer" is a written out answer.148 - "yes_no" is true iff the answer is Yes, and false iff the answer is No.149 150"evidence" is the set of paragraphs, figures or tables used to arrive at the answer. Tables or figures start with the string "FLOAT SELECTED"151 152"highlighted_evidence" is the set of sentences the answer providers selected as evidence if they chose textual evidence. The text in the "evidence" field is a mapping from these sentences to the paragraph level. That is, if you see textual evidence in the "evidence" field, it is guaranteed to be entire paragraphs, while that is not the case with "highlighted_evidence".153 154 155### Data Splits156 157|                            | Train   | Valid |158| -----                      | ------ | ----- |159| Number of papers            |     888   |      281 |160| Number of questions            |     2593   |      1005 |161| Number of answers            |     2675   |      1764 |162 163## Dataset Creation164 165### Curation Rationale166 167[More Information Needed]168 169### Source Data170 171NLP papers: The full text of the papers is extracted from [S2ORC](https://huggingface.co/datasets/s2orc) (Lo et al., 2020)172 173#### Initial Data Collection and Normalization174 175[More Information Needed]176 177#### Who are the source language producers?178 179[More Information Needed]180 181### Annotations182 183[More Information Needed]184 185#### Annotation process186 187[More Information Needed]188 189#### Who are the annotators?190 191"The annotators are NLP practitioners, not192expert researchers, and it is likely that an expert193would score higher"194 195### Personal and Sensitive Information196 197[More Information Needed]198 199## Considerations for Using the Data200 201### Social Impact of Dataset202 203[More Information Needed]204 205### Discussion of Biases206 207[More Information Needed]208 209### Other Known Limitations210 211[More Information Needed]212 213## Additional Information214 215### Dataset Curators216 217Crowdsourced NLP practitioners218 219### Licensing Information220 221[CC BY 4.0](https://creativecommons.org/licenses/by/4.0)222 223### Citation Information224 225```226@inproceedings{Dasigi2021ADO,227  title={A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers},228  author={Pradeep Dasigi and Kyle Lo and Iz Beltagy and Arman Cohan and Noah A. Smith and Matt Gardner},229  year={2021}230}231```232 233### Contributions234 235Thanks to [@cceyda](https://github.com/cceyda) for adding this dataset.236