DinoStackAI/qasper-rag

1---2license: cc-by-4.03task_categories:4- question-answering5- text-retrieval6language:7- en8tags:9- rag10- qasper11- scientific-qa12- retrieval13size_categories:14- 10K<n<100K15configs:16- config_name: answers17  data_files:18  - split: train19    path: answers/train*20  - split: dev21    path: answers/dev*22  - split: test23    path: answers/test*24- config_name: corpus25  data_files:26  - split: train27    path: corpus/*28- config_name: qrels29  data_files:30  - split: train31    path: qrels/train*32  - split: dev33    path: qrels/dev*34  - split: test35    path: qrels/test*36- config_name: queries37  data_files:38  - split: train39    path: queries/train*40  - split: dev41    path: queries/dev*42  - split: test43    path: queries/test*44- config_name: retrieved_docs45  data_files:46  - split: train47    path: retrieved_docs/train-*48  - split: dev49    path: retrieved_docs/dev-*50  - split: test51    path: retrieved_docs/test-*52- config_name: top_ranked53  data_files:54  - split: train55    path: top_ranked/train*56  - split: dev57    path: top_ranked/dev*58  - split: test59    path: top_ranked/test*60dataset_info:61  config_name: retrieved_docs62  features:63  - name: query_id64    dtype: string65  - name: corpus_id66    dtype: string67  - name: rank68    dtype: int6469  - name: retrieval_score70    dtype: float6471  - name: is_relevant72    dtype: bool73  splits:74  - name: train75    num_bytes: 168142476    num_examples: 2098577  - name: dev78    num_bytes: 71263279    num_examples: 889480  - name: test81    num_bytes: 104875782    num_examples: 1308983  download_size: 75882384  dataset_size: 344281385---86 87# QASPER RAG88 89Dataset for Retrieval-Augmented Generation (RAG) based on [QASPER](https://huggingface.co/datasets/allenai/qasper).90 91## Structure92 93| Subset | Splits | Description |94|--------|--------|-------------|95| `corpus` | train (default) | Paper chunks (abstract + full-text paragraphs) shared across all query splits |96| `queries` | train, dev, test | Information-seeking questions over scientific papers |97| `qrels` | train, dev, test | Relevance judgments (query ↔ paragraph chunk) |98| `answers` | train, dev, test | Reference answers (longest valid free-form answer) |99| `top_ranked` | train, dev, test | Paper-scoped candidate pool (all chunks of the query paper) |100| `retrieved_docs` | train, dev, test | Top-k retrieval results with relevance labels |101 102## Dataset statistics103 104| Split | Queries | Corpus |105|-------|--------:|-------:|106| train | 2101 | 81550 |107| dev   | 890 | 81550 |108| test  | 1310 | 81550 |109 110The corpus is shared across all splits and contains paragraph-level chunks from the abstract and full text of each paper.111 112- **Dev split:** mapped from the original `validation` split113- **Corpus source:** unique papers from train, validation and test splits114- **Chunking:** one chunk for the abstract (`section_name: abstract`) and one chunk per paragraph in `full_text`115 116## Source117 118| Component | QASPER resource |119|-----------|-----------------|120| Train | `train` split from [allenai/qasper](https://huggingface.co/datasets/allenai/qasper) |121| Dev | `validation` split |122| Test | `test` split |123| Corpus | `abstract` + `full_text.paragraphs` |124| Queries | `qas.question` |125| Qrels | `qas.answers[*].answer[*].evidence` matched to corpus chunks |126| Top ranked | All paragraph chunks from the query paper (paper-scoped retrieval pool) |127| Answers | Longest valid answer per question (`free_form_answer`, joined `extractive_spans`, or `Yes`/`No`) |128 129## Filtering130 131Questions are kept only when at least one answer satisfies all of the following:132 133- `unanswerable` is `false`134- the answer has `free_form_answer`, non-empty `extractive_spans`, or `yes_no`135- after removing evidence items containing `FLOAT SELECTED`, at least one answer still has evidence that matches the corpus136 137Evidence items containing `FLOAT SELECTED` are removed individually. Questions are omitted only when no valid answer has remaining evidence.138 139## Schema140 141### corpus142```json143{"id": "...", "title": "...", "section_name": "...", "text": "..."}144```145 146### queries147```json148{"id": "...", "text": "..."}149```150 151### qrels152```json153{"query_id": "...", "corpus_id": "...", "score": 1}154```155 156### answers157```json158{"query_id": "...", "answer": "..."}159```160 161### top_ranked162```json163{"query-id": "...", "corpus-ids": ["paper_00000", "paper_00001"]}164```165 166### retrieved_docs167```json168{"query_id": "...", "corpus_id": "...", "rank": 1, "retrieval_score": 0.92, "is_relevant": true}169```170 171Top-k documents retrieved from the indexed corpus (`is_relevant` is derived from `qrels`).172 173## Usage174 175```python176from datasets import load_dataset177 178corpus = load_dataset("DinoStackAI/qasper-rag", "corpus")["train"]179queries = load_dataset("DinoStackAI/qasper-rag", "queries")180qrels = load_dataset("DinoStackAI/qasper-rag", "qrels")181answers = load_dataset("DinoStackAI/qasper-rag", "answers")182 183train_queries = queries["train"]184dev_qrels = qrels["dev"]185test_answers = answers["test"]186```187 188## Citation189 190QASPER is released under the [CC BY 4.0 License](https://creativecommons.org/licenses/by/4.0/).191 192```bibtex193@inproceedings{Dasigi2021ADO,194  title={A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers},195  author={Pradeep Dasigi and Kyle Lo and Iz Beltagy and Arman Cohan and Noah A. Smith and Matt Gardner},196  year={2021}197}198```199