BeIR/cqadupstack

Dataset Card for BEIR Benchmark

Dataset Description

• —Homepage: https://beir.ai
• —Repository: https://beir.ai
• —Paper: https://openreview.net/forum?id=wCu6T5xFjeJ
• —Leaderboard: https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns
• —Point of Contact: nandan.thakur@uwaterloo.ca

Dataset Summary

BEIR is a heterogeneous benchmark built from 18 diverse datasets representing 9 information retrieval tasks.

• —Fact-checking: FEVER, Climate-FEVER, SciFact
• —Question-Answering: NQ, HotpotQA, FiQA-2018
• —Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
• —News Retrieval: TREC-NEWS, Robust04
• —Argument Retrieval: Touche-2020, ArguAna
• —Duplicate Question Retrieval: Quora, CqaDupstack
• —Citation-Prediction: SCIDOCS
• —Tweet Retrieval: Signal-1M
• —Entity Retrieval: DBPedia

This `cqadupstack` subset is the Duplicate Question Retrieval task within BEIR, with 12 domains.

Languages

All tasks are in English (en).

Dataset Structure

This dataset uses the standard BEIR retrieval layout with 12 domain configs: android, english, gaming, gis, mathematica, physics, programmers, stats, tex, unix, webmasters, wordpress.

Each config includes:

• —corpus: one row per document with _id, title, text
• —queries: one row per query with _id, title, text

Data Fields

• —_id (string): unique identifier
• —title (string): title (empty string when unavailable)
• —text (string): document/query text

CQADupstack Data Splits

Citation Information

@inproceedings{
thakur2021beir,
title={{BEIR}: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models},
author={Nandan Thakur and Nils Reimers and Andreas R{\"u}ckl{\'e} and Abhishek Srivastava and Iryna Gurevych},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
year={2021},
url={https://openreview.net/forum?id=wCu6T5xFjeJ}
}