PaDaS-Lab/CoRE

<h1 align="center">CoRE: Controlled Retrieval Evaluation Dataset</h1> <h4 align="center"> <p> <a href=#🔍-motivation>Motivation</a> | <a href=#📦-dataset-overview>Dataset Overview</a> | <a href=#🏗-dataset-construction>Dataset Construction</a> | <a href="#🧱-dataset-structure">Dataset Structure</a> | <a href="#🏷-qrels-format">Qrels Format</a> | <a href="#📊-evaluation">Evaluation</a> | <a href="#📜-citation">Citation</a> | <a href="#🔗-links">Links</a> | <a href="#📬-contact">Contact</a> <p> </h4>

CoRE (Controlled Retrieval Evaluation) is a benchmark dataset designed for the rigorous evaluation of embedding compression techniques in information retrieval. <!-- It isolates and controls critical variables—query relevance, distractor density, corpus size, and document length—to facilitate meaningful comparisons of retrieval performance across different embedding configurations. -->

🔍 Motivation

Embedding compression is essential for scaling modern retrieval systems, but its effects are often evaluated under overly simplistic conditions. CoRE addresses this by offering a collection of corpora with:

• —Multiple document lengths (passage and document) and sizes (10k to 100M)
• —Fixed number of relevant and distractor documents per query
• —Realistic evaluation grounded in TREC DL human relevance labels

This evaluation framework goes beyond, e.g., the benchmark used in the paper "The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes" from Reimers and Gurevych (2021), which disregards different document lengths and employs a less advanced random sampling, hence creating a less realistic experimental setup.

📦 Dataset Overview

CoRE builds on MS MARCO v2 and introduces high-quality distractors using pooled system runs from TREC 2023 Deep Learning Track. We ensure consistent query difficulty across different corpus sizes and document types. This overcomes the limitations of randomly sampled corpora, which can lead to trivial retrieval tasks, as no distractors are present in smaller datasets.

For each query:

• —10 relevant documents
• —100 high-quality distractors, selected via Reciprocal Rank Fusion (RRF) from top TREC system runs (bottom 20% of runs excluded)

🏗 Dataset Construction

To avoid trivializing the retrieval task when reducing corpus size, CoRE follows the intelligent corpus subsampling strategy proposed by Fröbe et al. (2025). This method is used to mine distractors from pooled ranking lists. These distractors are then included in all corpora of CoRE, ensuring a fixed query difficulty—unlike naive random sampling, where the number of distractors would decrease with corpus size.

Steps for both passage and document retrieval:

1. 1.Start from MS MARCO v2 annotations
2. 2.For each query:

• —Retain 10 relevant documents
• —Mine 100 distractors from RRF-fused rankings of top TREC 2023 DL submissions
• —Construct multiple corpus scales by aggregating relevant documents and distractors with randomly sampled filler documents

🧱 Dataset Structure

The dataset consists of three subsets: queries, qrels, and corpus.

• —queries: contains only one split (test)
• —qrels: contains two splits: passage and document
• —corpus: contains 11 splits, detailed below:

<div style="display: flex; gap: 2em;">

<table> <caption><strong>Passage Corpus Splits</strong></caption> <thead><tr><th>Split</th><th># Documents</th></tr></thead> <tbody> <tr><td>passcore</td><td>~7,130</td></tr> <tr><td>pass10k</td><td>~2,870</td></tr> <tr><td>pass100k</td><td>90,000</td></tr> <tr><td>pass1M</td><td>900,000</td></tr> <tr><td>pass10M</td><td>9,000,000</td></tr> <tr><td>pass100M</td><td>90,000,000</td></tr> </tbody> </table>

<table> <caption><strong>Document Corpus Splits</strong></caption> <thead><tr><th>Split</th><th># Documents</th></tr></thead> <tbody> <tr><td>doccore</td><td>~6,030</td></tr> <tr><td>doc10k</td><td>~3,970</td></tr> <tr><td>doc100k</td><td>90,000</td></tr> <tr><td>doc1M</td><td>900,000</td></tr> <tr><td>doc_10M</td><td>9,000,000</td></tr> </tbody> </table>

</div>

Note: The _core splits contain only relevant and distractor documents. All other splits are topped up with randomly sampled documents to reach the target size.

🏷 Qrels Format

The qrels files in CoRE differ from typical IR datasets. Instead of the standard relevance grading (e.g., 0, 1, 2), CoRE uses two distinct labels:

• —relevant (10 documents per query)
• —distractor (100 documents per query)

This enables focused evaluation of model sensitivity to compression under tightly controlled relevance and distractor distributions.

📊 Evaluation

from datasets import load_dataset

# Load queries
queries = load_dataset("PaDaS-Lab/CoRE", name="queries", split="test")

# Load relevance judgments
qrels = load_dataset("PaDaS-Lab/CoRE", name="qrels", split="passage")

# Load a 100k-scale corpus for passage retrieval
corpus = load_dataset("PaDaS-Lab/CoRE", name="corpus", split="pass_100k")

📜 Citation

If you use CoRE in your research, please cite:

@misc{caspari2025corect,
  title={CoRECT: A Framework for Evaluating Embedding Compression Techniques at Scale}, 
  author={L. Caspari and M. Dinzinger and K. Ghosh Dastidar and C. Fellicious and J. Mitrović and M. Granitzer},
  year={2025},
  eprint={2510.19340},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2510.19340}, 
}

🔗 Links

• —Paper
• —MS MARCO
• —TREC

📬 Contact

For questions or collaboration opportunities, contact us at michael.dinzinger@uni-passau.de.