BeIR/cqadupstack

1---2language:3- en4license:5- cc-by-sa-4.06multilinguality:7- monolingual8paperswithcode_id: beir9pretty_name: BEIR Benchmark10task_categories:11- zero-shot-classification12- text-retrieval13task_ids:14- document-retrieval15- entity-linking-retrieval16- fact-checking-retrieval17tags:18- biomedical-information-retrieval19- citation-prediction-retrieval20- passage-retrieval21- news-retrieval22- argument-retrieval23- zero-shot-information-retrieval24- tweet-retrieval25- question-answering-retrieval26- duplication-question-retrieval27- zero-shot-retrieval28configs:29- config_name: android30  data_files:31  - split: corpus32    path: android/corpus/corpus-*33  - split: queries34    path: android/queries/queries-*35- config_name: english36  data_files:37  - split: corpus38    path: english/corpus/corpus-*39  - split: queries40    path: english/queries/queries-*41- config_name: gaming42  data_files:43  - split: corpus44    path: gaming/corpus/corpus-*45  - split: queries46    path: gaming/queries/queries-*47- config_name: gis48  data_files:49  - split: corpus50    path: gis/corpus/corpus-*51  - split: queries52    path: gis/queries/queries-*53- config_name: mathematica54  data_files:55  - split: corpus56    path: mathematica/corpus/corpus-*57  - split: queries58    path: mathematica/queries/queries-*59- config_name: physics60  data_files:61  - split: corpus62    path: physics/corpus/corpus-*63  - split: queries64    path: physics/queries/queries-*65- config_name: programmers66  data_files:67  - split: corpus68    path: programmers/corpus/corpus-*69  - split: queries70    path: programmers/queries/queries-*71- config_name: stats72  data_files:73  - split: corpus74    path: stats/corpus/corpus-*75  - split: queries76    path: stats/queries/queries-*77- config_name: tex78  data_files:79  - split: corpus80    path: tex/corpus/corpus-*81  - split: queries82    path: tex/queries/queries-*83- config_name: unix84  data_files:85  - split: corpus86    path: unix/corpus/corpus-*87  - split: queries88    path: unix/queries/queries-*89- config_name: webmasters90  data_files:91  - split: corpus92    path: webmasters/corpus/corpus-*93  - split: queries94    path: webmasters/queries/queries-*95- config_name: wordpress96  data_files:97  - split: corpus98    path: wordpress/corpus/corpus-*99  - split: queries100    path: wordpress/queries/queries-*101dataset_info:102- config_name: android103  features:104  - name: _id105    dtype: string106  - name: title107    dtype: string108  - name: text109    dtype: string110  splits:111  - name: corpus112    num_bytes: 8255816113    num_examples: 22998114  - name: queries115    num_bytes: 31997116    num_examples: 699117  download_size: 8287813118  dataset_size: 8287813119- config_name: english120  features:121  - name: _id122    dtype: string123  - name: title124    dtype: string125  - name: text126    dtype: string127  splits:128  - name: corpus129    num_bytes: 12417951130    num_examples: 40221131  - name: queries132    num_bytes: 68095133    num_examples: 1570134  download_size: 12486046135  dataset_size: 12486046136- config_name: gaming137  features:138  - name: _id139    dtype: string140  - name: title141    dtype: string142  - name: text143    dtype: string144  splits:145  - name: corpus146    num_bytes: 14156689147    num_examples: 45301148  - name: queries149    num_bytes: 70040150    num_examples: 1595151  download_size: 14226729152  dataset_size: 14226729153- config_name: gis154  features:155  - name: _id156    dtype: string157  - name: title158    dtype: string159  - name: text160    dtype: string161  splits:162  - name: corpus163    num_bytes: 21024580164    num_examples: 37637165  - name: queries166    num_bytes: 39653167    num_examples: 885168  download_size: 21064233169  dataset_size: 21064233170- config_name: mathematica171  features:172  - name: _id173    dtype: string174  - name: title175    dtype: string176  - name: text177    dtype: string178  splits:179  - name: corpus180    num_bytes: 10765498181    num_examples: 16705182  - name: queries183    num_bytes: 34583184    num_examples: 804185  download_size: 10800081186  dataset_size: 10800081187- config_name: physics188  features:189  - name: _id190    dtype: string191  - name: title192    dtype: string193  - name: text194    dtype: string195  splits:196  - name: corpus197    num_bytes: 18811650198    num_examples: 38316199  - name: queries200    num_bytes: 47465201    num_examples: 1039202  download_size: 18859115203  dataset_size: 18859115204- config_name: programmers205  features:206  - name: _id207    dtype: string208  - name: title209    dtype: string210  - name: text211    dtype: string212  splits:213  - name: corpus214    num_bytes: 20111039215    num_examples: 32176216  - name: queries217    num_bytes: 41633218    num_examples: 876219  download_size: 20152672220  dataset_size: 20152672221- config_name: stats222  features:223  - name: _id224    dtype: string225  - name: title226    dtype: string227  - name: text228    dtype: string229  splits:230  - name: corpus231    num_bytes: 25668372232    num_examples: 42269233  - name: queries234    num_bytes: 30347235    num_examples: 652236  download_size: 25698719237  dataset_size: 25698719238- config_name: tex239  features:240  - name: _id241    dtype: string242  - name: title243    dtype: string244  - name: text245    dtype: string246  splits:247  - name: corpus248    num_bytes: 44756206249    num_examples: 68184250  - name: queries251    num_bytes: 117655252    num_examples: 2906253  download_size: 44873861254  dataset_size: 44873861255- config_name: unix256  features:257  - name: _id258    dtype: string259  - name: title260    dtype: string261  - name: text262    dtype: string263  splits:264  - name: corpus265    num_bytes: 25746785266    num_examples: 47382267  - name: queries268    num_bytes: 47358269    num_examples: 1072270  download_size: 25794143271  dataset_size: 25794143272- config_name: webmasters273  features:274  - name: _id275    dtype: string276  - name: title277    dtype: string278  - name: text279    dtype: string280  splits:281  - name: corpus282    num_bytes: 7353229283    num_examples: 17405284  - name: queries285    num_bytes: 24033286    num_examples: 506287  download_size: 7377262288  dataset_size: 7377262289- config_name: wordpress290  features:291  - name: _id292    dtype: string293  - name: title294    dtype: string295  - name: text296    dtype: string297  splits:298  - name: corpus299    num_bytes: 27597725300    num_examples: 48605301  - name: queries302    num_bytes: 24475303    num_examples: 541304  download_size: 27622200305  dataset_size: 27622200306---307 308# Dataset Card for BEIR Benchmark309 310## Dataset Description311 312- **Homepage:** https://beir.ai313- **Repository:** https://beir.ai314- **Paper:** https://openreview.net/forum?id=wCu6T5xFjeJ315- **Leaderboard:** https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns316- **Point of Contact:** nandan.thakur@uwaterloo.ca317 318### Dataset Summary319 320BEIR is a heterogeneous benchmark built from 18 diverse datasets representing 9 information retrieval tasks.321 322- Fact-checking: [FEVER](http://fever.ai), [Climate-FEVER](http://climatefever.ai), [SciFact](https://github.com/allenai/scifact)323- Question-Answering: [NQ](https://ai.google.com/research/NaturalQuestions), [HotpotQA](https://hotpotqa.github.io), [FiQA-2018](https://sites.google.com/view/fiqa/)324- Bio-Medical IR: [TREC-COVID](https://ir.nist.gov/covidSubmit/index.html), [BioASQ](http://bioasq.org), [NFCorpus](https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/)325- News Retrieval: [TREC-NEWS](https://trec.nist.gov/data/news2019.html), [Robust04](https://trec.nist.gov/data/robust/04.guidelines.html)326- Argument Retrieval: [Touche-2020](https://webis.de/events/touche-20/shared-task-1.html), [ArguAna](tp://argumentation.bplaced.net/arguana/data)327- Duplicate Question Retrieval: [Quora](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs), [CqaDupstack](http://nlp.cis.unimelb.edu.au/resources/cqadupstack/)328- Citation-Prediction: [SCIDOCS](https://allenai.org/data/scidocs)329- Tweet Retrieval: [Signal-1M](https://research.signal-ai.com/datasets/signal1m-tweetir.html)330- Entity Retrieval: [DBPedia](https://github.com/iai-group/DBpedia-Entity/)331 332> **This `cqadupstack` subset is the Duplicate Question Retrieval task within BEIR, with 12 domains.**333 334### Languages335 336All tasks are in English (`en`).337 338## Dataset Structure339 340This dataset uses the standard BEIR retrieval layout with 12 domain configs:341`android`, `english`, `gaming`, `gis`, `mathematica`, `physics`, `programmers`, `stats`, `tex`, `unix`, `webmasters`, `wordpress`.342 343Each config includes:344 345- `corpus`: one row per document with `_id`, `title`, `text`346- `queries`: one row per query with `_id`, `title`, `text`347 348### Data Fields349 350- `_id` (`string`): unique identifier351- `title` (`string`): title (empty string when unavailable)352- `text` (`string`): document/query text353 354### CQADupstack Data Splits355 356| Domain | Split | Rows |357| --- | --- | ---: |358| `android` | corpus | 22,998 |359| `android` | queries | 699 |360| `english` | corpus | 40,221 |361| `english` | queries | 1,570 |362| `gaming` | corpus | 45,301 |363| `gaming` | queries | 1,595 |364| `gis` | corpus | 37,637 |365| `gis` | queries | 885 |366| `mathematica` | corpus | 16,705 |367| `mathematica` | queries | 804 |368| `physics` | corpus | 38,316 |369| `physics` | queries | 1,039 |370| `programmers` | corpus | 32,176 |371| `programmers` | queries | 876 |372| `stats` | corpus | 42,269 |373| `stats` | queries | 652 |374| `tex` | corpus | 68,184 |375| `tex` | queries | 2,906 |376| `unix` | corpus | 47,382 |377| `unix` | queries | 1,072 |378| `webmasters` | corpus | 17,405 |379| `webmasters` | queries | 506 |380| `wordpress` | corpus | 48,605 |381| `wordpress` | queries | 541 |382 383## Citation Information384 385```bibtex386@inproceedings{387thakur2021beir,388title={{BEIR}: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models},389author={Nandan Thakur and Nils Reimers and Andreas R{\"u}ckl{\'e} and Abhishek Srivastava and Iryna Gurevych},390booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},391year={2021},392url={https://openreview.net/forum?id=wCu6T5xFjeJ}393}394```395