Datasetpublic

Hyukkyu/beir-scifact

SciFact — BEIR, unified schema A normalised copy of the dataset behind the mteb task SciFact, one of the tasks of the BEIR benchmark as mteb defines it. Same queries, documents and relevance judgements as the benchmark evaluates — reshaped into one strict schema shared by every dataset in this collection. Source mteb/scifact @ d56462d0e63a (the revision pinned in mteb) Domain · languages scientific · eng Queries / documents / qrels (all splits) 1,109 / 5,183 / 1… See the full description on the dataset page: https://huggingface.co/datasets/Hyukkyu/beir-scifact.

sourceHugging Facecc-by-nc-4.0updated 18d agoView on Hugging Face

0likes232downloads

Dataset Card

SciFact — BEIR, unified schema

A normalised copy of the dataset behind the mteb task `SciFact`, one of the tasks of the BEIR benchmark as mteb defines it. Same queries, documents and relevance judgements as the benchmark evaluates — reshaped into one strict schema shared by every dataset in this collection.


Source	`mteb/scifact` @ `d56462d0e63a` (the revision pinned in `mteb`)
Domain · languages	scientific · eng
Queries / documents / qrels (all splits)	1,109 / 5,183 / 1,258
Qrels per query	min 1 · mean 1.13 · max 5
Score values	1 ×339
Layout	`queries` · `corpus` · `qrels`, split `test`; `queries`/`qrels` also carry `train` — one shared corpus
Splits	`qrels`: train, test · `queries`: train, test
License	`cc-by-nc-4.0`

Schema

config	columns	rules
`queries`	`id: string`, `text: string`	ids unique and non-empty; every query has ≥ 1 qrel
`corpus`	`id: string`, `title: string`, `text: string`	`title` is always present (`""` when the source has none)
`qrels`	`query-id: string`, `corpus-id: string`, `score: int32`	referential integrity to both tables; no duplicate pairs; no floats

Files are Parquet, sorted by id, zstd-compressed, sharded at 500 MB. Every rule above is enforced by a validator before publishing; provenance.json records the source file hashes, what changed, and the output file hashes.

What changed from the source

—byte-preserved all text — no whitespace, newline, or control-character normalisation
—test: dropped 809 queries that have no qrels in this split (mteb does the same at load time)
—train: dropped 300 queries that have no qrels in this split (mteb does the same at load time)
—cast qrels.score double -> int32
—renamed _id → id
—renamed source splits (queries←queries/queries, corpus←corpus/corpus) to test

Load it

python

from datasets import load_dataset
queries = load_dataset("Hyukkyu/beir-scifact", "queries", split="test")
corpus  = load_dataset("Hyukkyu/beir-scifact", "corpus", split="test")
qrels   = load_dataset("Hyukkyu/beir-scifact", "qrels", split="test")

Cross-check against the previous layout

This repo previously held the same dataset in BEIR's raw two-repo form (Hyukkyu/beir-scifact + Hyukkyu/beir-scifact-qrels; that layout is kept under the git tag legacy-layout). Before replacing it, the build was compared against it: corpus id set identical (5,183 documents); qrels rows per split test 339 = 339, train 919 = 919.

License and attribution

The data is redistributed under the source's terms — cc-by-nc-4.0. All credit belongs to the original authors; see the source repository and the references in mteb's task metadata (https://github.com/allenai/scifact). This repository is an independent repackaging and is not affiliated with the RTEB or MTEB maintainers.