Hyukkyu/beir-fever

FEVER — BEIR, unified schema

A normalised copy of the dataset behind the mteb task `FEVER`, one of the tasks of the BEIR benchmark as mteb defines it. Same queries, documents and relevance judgements as the benchmark evaluates — reshaped into one strict schema shared by every dataset in this collection.

Schema

Files are Parquet, sorted by id, zstd-compressed, sharded at 500 MB. Every rule above is enforced by a validator before publishing; provenance.json records the source file hashes, what changed, and the output file hashes.

What changed from the source

• —byte-preserved all text — no whitespace, newline, or control-character normalisation
• —kept 3 qrels rows whose document is absent from the corpus — a property of the source (the source's qrels reference document ids that its corpus does not contain). mteb evaluates the source as-is, and removing them would change the ideal DCG and break equivalence with the benchmark, so they stay; the referential-integrity rule below is relaxed for this dataset only
• —dev: dropped 116,476 queries that have no qrels in this split (mteb does the same at load time)
• —test: dropped 116,476 queries that have no qrels in this split (mteb does the same at load time)
• —train: dropped 13,332 queries that have no qrels in this split (mteb does the same at load time)
• —cast qrels.score double -> int32
• —kept 31,703 documents whose text is identical to another document's — a property of the source; note that models embed such documents identically and retrieval ties among them are broken by corpus order
• —renamed _id → id
• —renamed source splits (queries←queries/queries, corpus←corpus/corpus) to test

Hard negatives and teacher scores

Filled by the owner's annotation pipeline (annotation=jina35) for the train split of the query set(s) below; queries without a labelled positive are left out.

• —Candidates: dense retrieval with jinaai/jina-embeddings-v5-text-small over the full corpus to depth 1,000; 100 candidates per query drawn from the rank windows 1–30 (30), 31–100 (30), 101–300 (20), 301–1000 (20), the query's labelled positives excluded. rank is the dense rank; source is dense for a mined row and dataset for a negative the source labels itself (those are kept for every query of the split, sampled or not).
• —Teacher: jinaai/jina-reranker-v3.5, listwise: a query's positive and all of its candidates are scored together in one context of up to 32,768 tokens. score is the raw cosine score, one row per (query, positive) and per (query, candidate); a labelled negative that was also mined is scored once. No filtering is applied to the tables.

from datasets import load_dataset
negatives = load_dataset("Hyukkyu/beir-fever", "hard-negatives", split="train")
scores    = load_dataset("Hyukkyu/beir-fever", "teacher-scores", split="train")

Load it

from datasets import load_dataset
queries = load_dataset("Hyukkyu/beir-fever", "queries", split="test")
corpus  = load_dataset("Hyukkyu/beir-fever", "corpus", split="test")
qrels   = load_dataset("Hyukkyu/beir-fever", "qrels", split="test")

Cross-check against the previous layout

This repo previously held the same dataset in BEIR's raw two-repo form (Hyukkyu/beir-fever + Hyukkyu/beir-fever-qrels; that layout is kept under the git tag legacy-layout). Before replacing it, the build was compared against it: corpus id set identical (5,416,568 documents); qrels rows per split test 7,937 = 7,937, train 140,085 = 140,085, dev 8,079 = 8,079.

License and attribution

The data is redistributed under the source's terms — cc-by-nc-sa-3.0. All credit belongs to the original authors; see the source repository and the references in mteb's task metadata. This repository is an independent repackaging and is not affiliated with the RTEB or MTEB maintainers.

License discrepancy. The upstream dataset card declares cc-by-nc-sa-3.0 while mteb's TaskMetadata declares None. This card carries the upstream value as the more conservative choice.