orgrctera/beir_scifact

BEIR SciFact (orgrctera/beir_scifact)

Overview

SciFact is an expert-annotated corpus for scientific claim verification: given a short scientific claim, systems must find PubMed abstracts in a fixed corpus that contain evidence supporting or refuting the claim. The original work frames the task as retrieval plus rationale labeling; BEIR (Benchmarking-IR) repurposes SciFact as a zero-shot information retrieval benchmark, where the goal is to rank abstracts so that relevant evidence (supporting or refuting) appears at the top.

This Hub release packages the SciFact split used in BEIR as a retrieval dataset: each row is one query (a scientific claim) paired with gold relevant document IDs from the SciFact corpus. The corpus itself is 5,183 PubMed abstracts; the BEIR-style query sets cover 1,109 claims (see Splits below).

Task

• —Task type: Retrieval (dense retrieval, sparse retrieval, re-ranking, or full RAG pipelines) on SciFact within the BEIR evaluation setting.
• —Input: A natural-language claim (input).
• —Supervision / reference: expected_output is a JSON string listing relevant corpus document IDs with a relevance score (here 1 for relevant), matching the BEIR qrels-style supervision used for ranking metrics.

Models are typically evaluated with standard IR metrics (e.g. nDCG@k, Recall@k) after retrieving from the SciFact abstract corpus, using the same query/qrels splits as BEIR. The full verification task (labeling SUPPORT / REFUTE / NO_INFO and extracting rationales) is defined in the original SciFact paper; BEIR focuses on whether the right abstracts are retrieved.

Background

SciFact

Wadden et al. (EMNLP 2020) introduced scientific claim verification: select abstracts from the literature that contain evidence that SUPPORTS or REFUTES a claim, and identify rationales within those abstracts. They released SciFact—about 1.4K expert-written biomedical claims paired with evidence-bearing abstracts and fine-grained annotations. The domain is highly specialized (synonyms, abbreviations, technical wording), so lexical overlap alone is often misleading; semantic retrieval is stressed.

• —Project / data: allenai/scifact on GitHub · SciFact dataset on Hugging Face

BEIR

Thakur et al. (2021) introduced BEIR: a heterogeneous benchmark for zero-shot evaluation of retrieval models across 18 public datasets and diverse domains. SciFact is one of the included datasets. For IR scoring, supports and refutes evidence are treated as equally relevant (binary relevance to the claim), as is standard in BEIR’s retrieval formulation.

• —Code / leaderboard context: UKPLab/beir

Relation to ir-datasets / beir/scifact

The same BEIR SciFact corpus and queries are documented in ir-datasets (e.g. docs/queries views). This dataset is a tabular export aligned with other CTERA benchmark releases: one row per query with string expected_output for tooling compatibility.

Data fields

Splits

Examples

Illustrative rows from this dataset (IDs and text as stored).

Example 1 — train

• —`input`: siRNA knockdown of A20 accelerates tumor progression in an in vivo murine xenograft model.
• —`expected_output`:

[{"id": "33370", "score": 1}, {"id": "38355793", "score": 1}]

• —`metadata.query_id`: 1403 · `metadata.split`: train

Example 2 — test

• —`input`: Hypothalamic glutamate neurotransmission is crucial to energy balance.
• —`expected_output`:

[{"id": "11886686", "score": 1}, {"id": "25007443", "score": 1}]

• —`metadata.query_id`: 540 · `metadata.split`: test

References

SciFact (original task and dataset)

Abstract (arXiv:2004.14974): We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that SUPPORTS or REFUTES a given scientific claim, and to identify rationales justifying each decision. To study this task, we construct SciFact, a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales.

• —Fact or Fiction: Verifying Scientific Claims — arXiv:2004.14974
• —ACL Anthology (EMNLP 2020)

BEIR (retrieval benchmark including SciFact)

Abstract (arXiv:2104.08663): We introduce Benchmarking-IR (BEIR), a robust and heterogeneous evaluation benchmark for information retrieval. We leverage a careful selection of 18 publicly available datasets from diverse text retrieval tasks and domains…

• —BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — arXiv:2104.08663

Citation

If you use SciFact, cite:

@inproceedings{Wadden2020Scifact,
  title = {Fact or Fiction: Verifying Scientific Claims},
  author = {Wadden, David and Lin, Shanchuan and Lo, Kyle and Wang, Lucy Lu and
            van Zuylen, Madeleine and Cohan, Arman and Hajishirzi, Hannaneh},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  month = nov,
  year = {2020},
  address = {Online},
  publisher = {Association for Computational Linguistics},
  url = {https://www.aclweb.org/anthology/2020.emnlp-main.609},
  doi = {10.18653/v1/2020.emnlp-main.609},
  pages = {7534--7550}
}

If you use the BEIR benchmark formulation, cite:

@article{Thakur2021Beir,
  title = {{BEIR}: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models},
  author = {Thakur, Nandan and Reimers, Nils and R{\"u}ckl{\'e}, Andreas and
            Srivastava, Abhishek and Gurevych, Iryna},
  journal = {arXiv preprint arXiv:2104.08663},
  year = {2021},
  url = {https://arxiv.org/abs/2104.08663}
}

Provenance

Exported for retrieval evaluation (e.g. Langfuse / internal tooling) with SciFact as the BEIR sub-benchmark scifact. Corpus documents are not duplicated in every row; join expected_output document IDs to the SciFact corpus for full abstract text when building an index.