isaacus/gdpr-holdings-retrieval

GDPR Holdings Retrieval 🔐

GDPR Holdings Retrieval by Isaacus is a novel and challenging legal information retrieval evaluation dataset consisting of 500 fact patterns paired with holdings) in European regulatory and court decisions.

This dataset is intended to stress test the ability of an information retrieval model to retrieve relevant judicial and regulatory decisions given arbitrary fact patterns.

This dataset forms part of the Massive Legal Embeddings Benchmark (MLEB), the largest, most diverse, and most comprehensive benchmark for legal text embedding models.

Structure 🗂️

As per the MTEB information retrieval dataset format, this dataset comprises three splits, default, corpus and queries.

The default split pairs fact patterns (query-id) with relevant holding summaries (corpus-id), each pair having a score of 1.

The corpus split contains holding summaries, with the text of such summaries being stored in the text key and their ids being stored in the _id key. There is also a title column which is deliberately set to an empty string in all cases for compatibility with the `mteb` library.

The queries split contains fact patterns, with the text of such fact patterns being stored in the text key and their ids being stored in the _id key.

Methodology 🧪

This dataset was constructed by collecting all GDPRHub articles, using regex to separate their facts and holdings sections and then converting those sections into plain text with Inscriptis, sampling 500 pairs for inclusion in this dataset.

License 📜

This dataset is licensed under the same license as GDPRHub, CC BY NC SA 4.0, which permits non-commercial use only of this dataset as long as appropriate attribution is made and derivative works are issued under the same license.

Citation 🔖

If you use this dataset, please cite the Massive Legal Embeddings Benchmark (MLEB):

@misc{butler2025massivelegalembeddingbenchmark,
      title={The Massive Legal Embedding Benchmark (MLEB)}, 
      author={Umar Butler and Abdur-Rahman Butler and Adrian Lucas Malec},
      year={2025},
      eprint={2510.19365},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.19365}, 
}