Datasetpublic

Hyukkyu/beir-hotpotqa

HotpotQA — BEIR, unified schema A normalised copy of the dataset behind the mteb task HotpotQA, one of the tasks of the BEIR benchmark as mteb defines it. Same queries, documents and relevance judgements as the benchmark evaluates — reshaped into one strict schema shared by every dataset in this collection. Source mteb/hotpotqa @ ab518f4d6fcc (the revision pinned in mteb) Domain · languages wikipedia · eng Queries / documents / qrels (all splits) 97,852 / 5,233… See the full description on the dataset page: https://huggingface.co/datasets/Hyukkyu/beir-hotpotqa.

sourceHugging Facecc-by-sa-4.0updated 10d agoView on Hugging Face

0likes130downloads

Dataset Card

HotpotQA — BEIR, unified schema

A normalised copy of the dataset behind the mteb task `HotpotQA`, one of the tasks of the BEIR benchmark as mteb defines it. Same queries, documents and relevance judgements as the benchmark evaluates — reshaped into one strict schema shared by every dataset in this collection.


Source	`mteb/hotpotqa` @ `ab518f4d6fcc` (the revision pinned in `mteb`)
Domain · languages	wikipedia · eng
Queries / documents / qrels (all splits)	97,852 / 5,233,329 / 195,704
Qrels per query	min 2 · mean 2.0 · max 2
Score values	1 ×14,810
Layout	`queries` · `corpus` · `qrels`, split `test`; `queries`/`qrels` also carry `train`, `dev` — one shared corpus; `hard-negatives` and `teacher-scores` for `train`
Splits	`hard-negatives`: train · `qrels`: train, dev, test · `queries`: train, dev, test · `teacher-scores`: train
Training extension	`hard-negatives` and `teacher-scores` for the `train` split are the owner's own mining and scoring (details); `queries`/`qrels` `train` are the benchmark's training data
Hard negatives	sources: `dense` · 8,366,684 rows
Teacher scores	`jinaai/jina-reranker-v3.5` · 8,451,684 rows (positives included)
License	`cc-by-sa-4.0`

Schema

config	columns	rules
`queries`	`id: string`, `text: string`	ids unique and non-empty; every query has ≥ 1 qrel
`corpus`	`id: string`, `title: string`, `text: string`	`title` is always present (`""` when the source has none)
`qrels`	`query-id: string`, `corpus-id: string`, `score: int32`	referential integrity to both tables; no duplicate pairs; no floats
`hard-negatives`	`query-id: string`, `corpus-id: string`, `rank: int32`, `source: string`	one row per mined negative; `(query-id, corpus-id, source)` unique; never a labelled positive of the same query
`teacher-scores`	`query-id: string`, `corpus-id: string`, `teacher: string`, `score: float32`	one row per scored pair (positives included); a row means scored — never a placeholder

Files are Parquet, sorted by id, zstd-compressed, sharded at 500 MB. Every rule above is enforced by a validator before publishing; provenance.json records the source file hashes, what changed, and the output file hashes.

What changed from the source

—byte-preserved all text — no whitespace, newline, or control-character normalisation
—dev: dropped 92,405 queries that have no qrels in this split (mteb does the same at load time)
—test: dropped 90,447 queries that have no qrels in this split (mteb does the same at load time)
—train: dropped 12,852 queries that have no qrels in this split (mteb does the same at load time)
—cast qrels.score double -> int32
—kept 12,694 documents whose text is identical to another document's — a property of the source; note that models embed such documents identically and retrieval ties among them are broken by corpus order
—renamed _id → id
—renamed source splits (queries←queries/queries, corpus←corpus/corpus) to test

Hard negatives and teacher scores

Filled by the owner's annotation pipeline (annotation=jina35) for the train split of the query set(s) below; queries without a labelled positive are left out.

—Candidates: dense retrieval with jinaai/jina-embeddings-v5-text-small over the full corpus to depth 1,000; 100 candidates per query drawn from the rank windows 1–30 (30), 31–100 (30), 101–300 (20), 301–1000 (20), the query's labelled positives excluded. rank is the dense rank; source is dense for a mined row and dataset for a negative the source labels itself (those are kept for every query of the split, sampled or not).
—Teacher: jinaai/jina-reranker-v3.5, listwise: a query's positive and all of its candidates are scored together in one context of up to 32,768 tokens. score is the raw cosine score, one row per (query, positive) and per (query, candidate); a labelled negative that was also mined is scored once. No filtering is applied to the tables.

configs	queries	hard negatives	teacher scores
`hard-negatives` · `teacher-scores`	85,000 (seeded sample, seed 1)	8,366,684 (8,366,684 dense)	8,451,684

python

from datasets import load_dataset
negatives = load_dataset("Hyukkyu/beir-hotpotqa", "hard-negatives", split="train")
scores    = load_dataset("Hyukkyu/beir-hotpotqa", "teacher-scores", split="train")

Load it

python

from datasets import load_dataset
queries = load_dataset("Hyukkyu/beir-hotpotqa", "queries", split="test")
corpus  = load_dataset("Hyukkyu/beir-hotpotqa", "corpus", split="test")
qrels   = load_dataset("Hyukkyu/beir-hotpotqa", "qrels", split="test")

Cross-check against the previous layout

This repo previously held the same dataset in BEIR's raw two-repo form (Hyukkyu/beir-hotpotqa + Hyukkyu/beir-hotpotqa-qrels; that layout is kept under the git tag legacy-layout). Before replacing it, the build was compared against it: corpus id set identical (5,233,329 documents); qrels rows per split test 14,810 = 14,810, train 170,000 = 170,000, dev 10,894 = 10,894.

License and attribution

The data is redistributed under the source's terms — cc-by-sa-4.0. All credit belongs to the original authors; see the source repository and the references in mteb's task metadata. This repository is an independent repackaging and is not affiliated with the RTEB or MTEB maintainers.

License discrepancy. The upstream dataset card declares cc-by-sa-4.0 while mteb's TaskMetadata declares None. This card carries the upstream value as the more conservative choice.