AccountVerify/legalpincite

👩‍⚖️ LegalPincite: Multi-level Legal Information Retrieval Dataset

<p align="center"> <img src=https://huggingface.co/datasets/theresiavr/legalpincite/resolve/main/legalpincite_overview.png alt="LegalPincite Overview. Credits: pin icon created by alfanz - Flaticon; Lawyer icon created by Magnific - Flaticon"/> </p>

LegalPincite is a large-scale legal information retrieval (IR) test collection built from Court of Justice of the European Union (CJEU) judgments in EUR-Lex. It is designed for citation-oriented legal retrieval at multiple levels of granularity: case-to-case retrieval, paragraph-to-case retrieval, and paragraph-to-paragraph pinpoint citation (pincite) retrieval. The dataset is especially useful for evaluating systems that must retrieve the exact paragraph-level legal authority, that supports a citing legal paragraph.

The dataset addresses limitations in prior legal IR resources by providing masked query text, document text (corpora) that include both cited and non-cited paragraphs to search on, and ground-truth citation links at both case and paragraph levels.

• —Dataset repository: <https://huggingface.co/datasets/theresiavr/legalpincite>
• —Dataset DOI: <https://doi.org/10.57967/hf/9072>
• —Paper: <https://arxiv.org/abs/2608.03756>
• —Code/examples repository: <https://github.com/theresiavr/legalpincite>
• —Dataset sources:
• —Primary source: <https://eur-lex.europa.eu/>
• —Ground truth citation up to 2021 (Olsen et al., 2023): <https://huggingface.co/datasets/ngarneau/paragraphtoparagraph>
• —Expert annotation data (Olsen et al., 2023): <https://github.com/coastalcph/paragraph_network/tree/main/coding>
• —Document text up to 2024: previously unreleased dataset by Olsen et al. (2026)

Important: This dataset is for research and benchmarking. It should not be treated as legal advice or a complete representation of CJEU precedent.

Table of Contents

1. 1.Dataset Details
2. 2.Intended Usage
3. 3.Dataset Structure
4. 4.Loading the Dataset
5. 5.Dataset Creation
6. 6.Statistics
7. 7.Evaluation Guidance
8. 8.Biases, Risks, and Limitations
9. 9.Ethical and Legal Considerations
10. 10.Citation
11. 11.Acknowledgment
12. 12.Glossary

Dataset Details

LegalPincite is a multi-level legal IR dataset composed of CJEU judgments and citation relationships extracted from EUR-Lex. It supports three retrieval settings:

The paragraph-to-paragraph setting is the most fine-grained task. It reflects a common legal practice in which an opinion, judgment, or brief cites not only another case, but a specific paragraph within that case.

LegalPincite provides:

• —*Masked queries (in query)* in which citation-bearing information has been removed or anonymized to reduce data leakage.
• —*Unmasked queries (in query)* for transparency and ablation studies. These should not be used for normal retrieval evaluation.
• —*Candidate corpora (in doc)* that include both cited and non-cited paragraphs, making the retrieval setting more realistic than previous collection (Olsen et al., 2023) that contains only positive targets.
• —*Ground truth citations (in qrel)* information on which cases/paragraphs are cited or are relevant for case-case, paragraph-case, and paragraph-paragraph retrieval.
• —*Ground-truth sources (in qrel)* indicating whether relevance indicator originate from EUR-Lex citations or expert human annotation.

Dataset Metadata

Intended Usage

Direct Use

LegalPincite is intended for research and development in legal information retrieval and related NLP tasks. Suitable uses include:

• —Benchmarking sparse, dense, hybrid, neural, and reranking-based retrieval systems.
• —Evaluating citation retrieval systems at case and paragraph level.
• —Developing methods for legal pincite recommendation.
• —Training supervised legal ranking or retrieval models using the train query and qrels.
• —Studying temporal retrieval settings in which legal precedents must predate the query case.
• —Analysing citation networks and paragraph-level legal authority relationships.

Secondary and Research Extensions

With additional processing, the dataset may support:

• —Link prediction over legal citation networks.
• —Legal textual entailment or rule-support identification.
• —Cross-lingual legal retrieval, if aligned document are collected from EUR-Lex in other EU languages.
• —Building and evaluating legal Retrieval-Augmented Generation (RAG) systems, especially the retrieval component.
• —Legal reasoning in Large Language Models (LLMs)

Out-of-Scope Use

The dataset is not suitable for:

• —Providing legal advice or making legal determinations.
• —Replacing professional legal research, legal review, or judicial reasoning.
• —Automated decision-making affecting individuals, institutions, rights, obligations, or access to services.
• —Claims about complete coverage of all relevant CJEU authorities for a legal issue.
• —Benchmarking recall-sensitive systems as if qrels were exhaustive judgments over all relevant documents.

Dataset Structure

LegalPincite is organized as 20 CSV files:

where * can be replaced by train/dev/test.

We also provide the sparse index (Terrier index) for the candidate cases and paragraphs across the two splits:

• —<https://huggingface.co/datasets/theresiavr/legalpincitedocdev_case.terrier>
• —<https://huggingface.co/datasets/theresiavr/legalpincitedoctest_case.terrier>
• —<https://huggingface.co/datasets/theresiavr/legalpincitedocdev_par.terrier>
• —<https://huggingface.co/datasets/theresiavr/legalpincitedoctest_par.terrier>

Our code repository shows how to load and use them for retrieval, as well as example code on how to re-create the index from the CSV doc_ files.

File Schemas

We describe the content of columns in each file type (query, document, qrel, and metadata)

• —split refers to train/dev/test
• —level refers to case or par (paragraph)

Query files: query_{split}_{level}.csv

Document files: doc_{split}_{level}.csv

Qrel files: qrel_{split}_{query_level}_{doc_level}.csv

Metadata file: metadata.csv

Identifier Conventions

• —Case-level IDs: CELEX identifiers, e.g. 62021CJ0326. CELEX refers to the unique, language-independent identification code assigned to European Union legal documents in the EUR-Lex Database.
• —Paragraph-level IDs: CELEX plus paragraph number, e.g. 62021CJ0326-42 where 42 is the paragraph number.
• —Qrels: The qid and docno columns use the appropriate case-level or paragraph-level identifier depending on the task.

Split Design

The dataset uses chronological splits to reduce temporal leakage and model a realistic legal retrieval scenario. A later case can cite earlier precedent, but an earlier case cannot cite a future case.

Note: Document/candidate files are provided only for dev and test. Train qrels are provided for supervised training or model development, but there is no separate doc_train_* candidate file. The doc_dev_* file can be used instead.

Loading the Dataset

We also provide end-to-end example usage in the code/examples repository: <https://github.com/theresiavr/legalpincite>

With datasets

from datasets import load_dataset

# Example config to load dataset for paragraph-to-paragraph retrieval (test split)
query_config = "query_par"
doc_config = "doc_par"
qrel_config = "qrel_par_par"
split = "test"

queries = load_dataset("theresiavr/legalpincite", query_config, split=split)
docs = load_dataset("theresiavr/legalpincite", doc_config, split=split)
qrels = load_dataset("theresiavr/legalpincite", qrel_config, split=split)

print(queries[0])
print(docs[0])
print(qrels[0])

With pandas

import pandas as pd

# Example script to load dataset for paragraph-to-paragraph retrieval (test split)

base = "https://huggingface.co/datasets/theresiavr/legalpincite/resolve/main"

queries = pd.read_csv(f"{base}/query_test_par.csv")
docs = pd.read_csv(f"{base}/doc_test_par.csv")
qrels = pd.read_csv(f"{base}/qrel_test_par_par.csv")

print(queries.head())
print(docs.head())
print(qrels.head())

Dataset Creation

Curation Rationale

Legal practice often requires citations to specific legal paragraphs rather than only to full cases. Existing public legal IR datasets tend to focus on case-level or paragraph-to-case retrieval and do not fully support paragraph-to-paragraph pincite retrieval. LegalPincite was created to fill this gap with a realistic corpus that includes non-cited paragraphs and masked queries.

Source Data

The dataset is built from CJEU judgments in EUR-Lex. It combines and corrects earlier legal IR resources, then expands coverage with more recent judgments.

The construction pipeline is based on the following datasets:

1. 1.An unpublished all-paragraph dataset based on Olsen et al. (2026), containing CJEU judgment paragraphs up to 29 July 2024. This source provides paragraph segmentations and candidate paragraph text. LegalPincite corrects 1,400+ cases with erroneous segmentation.
2. 2.A published citation dataset from Olsen et al. (2023), containing 110,601 citing-cited paragraph pairs from CJEU judgments up to 6 October 2021 and 890 relevance annotations by two law experts. LegalPincite retrieves the missing text of 268 paragraphs and makes the relevance annotations that are previously un-reusable due to missing IDs.
3. 3.A 2026 update, collecting CJEU judgments (in English) dated 1 January 2021 through 31 December 2025, resulting in 2,170 unique cases and 41,547 unique paragraph-to-paragraph citations before final exclusions and merging.

Data Collection and Processing

The dataset curators performed several quality-control and preprocessing steps:

• —Identified and removed non-English paragraphs using langdetect.
• —Recovered missing citing/cited paragraph text from the all-paragraph dataset and by consulting EUR-Lex manually where needed.
• —Corrected several citing/cited paragraph-number errors.
• —Removed paragraph pairs whose text could not be recovered.
• —Re-extracted HTML/XHTML source from EUR-Lex via the Cellar API where paragraph parsing errors affected earlier resources. This is done to adjust paragraph segmentation manually or automatically where consecutive paragraphs were joined.
• —Linked expert relevance annotations back to CELEX and paragraph identifiers.
• —Merged corrected existing data with updated 2021-2025 citation data and removed duplicates.
• —Removed trailing whitespace.
• —Aggregated paragraph-to-paragraph citations into paragraph-to-case and case-to-case qrels.
• —Created masked queries using a pretrained legal NER model (<https://huggingface.co/opennyaiorg/enlegalner_trf>) and regular expressions to remove citation-specific leakage cues. This is because some citing paragraphs explicitly mention the cited case, case number, parties, or target paragraph. If left unmasked, retrieval systems could exploit these textual clues rather than learn substantive legal relevance.

Annotations

Most qrels (ground truth citations) are derived from EUR-Lex citations, written by their respective authors (e.g., specifc judge assigned to the case). The dev split also includes expert relevance annotations from prior work. For expert annotations, a query-paragraph pair is treated as relevant if both expert annotators answered yes to at least one of these criteria:

1. 1.The candidate paragraph contains a verbatim version of the rule in the citing paragraph.
2. 2.The candidate paragraph contains a different or more expanded version of the rule in the citing paragraph.

Expert annotations were created by two human law experts as part of the earlier paragraph-level citation work (Olsen et al., 2023). Pairs considered irrelevant are not included in the qrel files.

Statistics

Query and Qrel Statistics

Candidate Corpus Statistics

Qrel Source Distribution

The train and test qrels are based on EUR-Lex citations. The dev qrels include both EUR-Lex-derived qrels and human expert annotations.

In the qrel file, "Both" pairs contribute two rows.

Evaluation Guidance

Recommended Metrics

LegalPincite is best evaluated with ranking metrics that emphasize early precision. As qrels are not exhaustive judgments over every potentially relevant legal authority, recall-oriented evaluation should be interpreted carefully.

Suggested Experimental Protocol

For standard benchmark experiments:

1. 1.Select a retrieval level: case_case, par_case, or par_par.
2. 2.Select a split (dev/test).
3. 3.Use the corresponding query file and document corpus.
4. 4.Optional: select a subset of query/qrels to be used (e.g., query/qrels from EUR-Lex, from expert annotation only, or any)
5. 5.Use the masked query column, not query_unmasked.
6. 6.Rank all candidate documents in the relevant doc_* split.
7. 7.Evaluate against the matching qrel file.

Example mapping:

Biases, Risks, and Limitations

Legal and Jurisdictional Scope

LegalPincite covers CJEU judgments and citations available through EUR-Lex. It does not represent all EU legal materials, national court decisions, legislation, administrative materials, legal scholarship, or legal practice across jurisdictions.

Citation-Derived Ground Truth

Most qrels are based on observed citations in judgments. A cited case or paragraph is relevant in the sense that it was cited by the court, but the absence of a citation does not prove irrelevance. Judicial citation practices are selective and may be influenced by legal convention, author preference, prior systems, and institutional practices.

Feedback Loop Risk

Because judges and legal professionals may use legal search tools when identifying authorities, citation-derived qrels can reflect existing retrieval systems or citation practices. This may create a feedback loop in which benchmark targets mirror prior search behavior.

Non-Exhaustive Human Annotations

The expert annotations were created over top-10 retrieved results from a dense retriever in prior work (Olsen et al., 2023). These annotations add value but are not exhaustive over the full corpus. LegalPincite should therefore be treated as a benchmark for ranking known relevant items highly, not as a complete legal relevance collection.

Information Leakage in Queries Is Mitigated, Not Eliminated

Query masking uses automatic NER and regular expressions. Some leakage cues may remain. Users should inspect examples and consider additional masking if a method appears to exploit case numbers, party names, dates, or citation strings.

Known Language Caveat

The dataset construction process includes automatic removal of non-English paragraphs. However, legal texts from EUR-Lex may contain multilingual fragments, and automatic filtering may miss some non-English content.

Long Documents and Computational Cost

CJEU cases are long. Case-level documents average thousands of words, and paragraph-level corpora contain nearly 600,000 candidate paragraphs. Dense retrieval, exhaustive reranking, and cross-encoder evaluation may require substantial compute. In this case, we recommend using a subset of the queries (e.g., the ones with human annotations)

Recommendations

• —Use query, not query_unmasked, for standard evaluation.
• —Prefer precision-oriented metrics at short cutoffs.
• —Report levels of query and qrel (doc/par), split, and qrel source handling.
• —Do not interpret missing qrels as true negative legal judgments.
• —Perform additional legal validation before using retrieved citations in any real legal workflow.
• —To reduce computational cost, use a subset of the queries (e.g., the ones with human annotations)

Ethical and Legal Considerations

Personal and Sensitive Information

CJEU judgments may contain names of individuals, organizations, public bodies, companies, case parties, and factual details. The dataset includes public legal texts from EUR-Lex, and citation masking may remove some names or entities in query text, but this is not a full anonymization or privacy-preserving transformation.

License and Source Terms

The dataset is released under Creative Commons Attribution 4.0 International (CC BY 4.0). Users are responsible for complying with the dataset license and attribution requirements.

Citation

If you use this dataset, please cite the dataset DOI, the accompanying paper, and the papers of the original datasets.

Dataset Citation

@dataset{rampisela_legalpincite_2026,
  title        = {LegalPincite: Multi-level Legal Information Retrieval Dataset},
  author       = {Rampisela, Theresia Veronika and Olsen, Henrik Palmer and Colavizza, Giovanni},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/9072},
  url          = {https://huggingface.co/datasets/theresiavr/legalpincite},
  license      = {CC-BY-4.0}
}

Paper Citation

Accompanying dataset paper

• —Theresia Veronika Rampisela, Henrik Palmer Olsen, and Giovanni Colavizza. 2026. LegalPincite: Multi-level Legal Information Retrieval Dataset. Preprint. <https://arxiv.org/abs/2608.03756>

@misc{rampisela2026legalpincite,
      title={LegalPincite: Multi-level Legal Information Retrieval Dataset}, 
      author={Theresia Veronika Rampisela and Henrik Palmer Olsen and Giovanni Colavizza},
      year={2026},
      eprint={2608.03756},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2608.03756}, 
}

Papers of the original datasets

• —Henrik Palmer Olsen, Nicolas Garneau, Yannis Panagis, and Johan Lindholm. 2026. Providing legal pincite recommendations using language representations. Artificial Intelligence and Law (2026). <doi:10.1007/s10506-025-09493-3>
• —Henrik Palmer Olsen, Nicolas Garneau, Yannis Panagis, Johan Lindholm, and Anders Søgaard. 2023. Re-Framing Case Law Citation Prediction from a Paragraph Perspective. In Legal Knowledge and Information Systems - JURIX 2023 (Frontiers in Artificial Intelligence and Applications). Giovanni Sileno, Jerry Spanakis, and Gijs van Dijck (Eds.). IOS Press BV, Netherlands, 323–328. <doi:10.3233/FAIA230982>

@inproceedings{Olsen2023ReFraming,
  title = "Re-Framing Case Law Citation Prediction from a Paragraph Perspective",
  author = "Olsen, {Henrik Palmer} and Nicolas Garneau and Yannis Panagis and Johan Lindholm and Anders S{\o}gaard",
  year = "2023",
  doi = "10.3233/FAIA230982",
  series = "Frontiers in Artificial Intelligence and Applications",
  publisher = "IOS Press BV",
  pages = "323--328",
  editor = "Giovanni Sileno and Jerry Spanakis and {van Dijck}, Gijs",
  booktitle = "Legal Knowledge and Information Systems - JURIX 2023",
  address = "Netherlands",
}

@article{Olsen2026ProvidingPincite,
  title = "Providing legal pincite recommendations using language representations",
  author = "Olsen, {Henrik Palmer} and Nicolas Garneau and Yannis Panagis and Johan Lindholm",
  year = "2026",
  doi = "10.1007/s10506-025-09493-3",
  journal = "Artificial Intelligence and Law",
  issn = "0924-8463",
  publisher = "Springer Nature",
}

Acknowledgment

We thank Yannis Panagis for creating the two original datasets (Olsen et al., 2023; Olsen et al., 2026) that served as the foundation for this dataset.

Glossary

Dataset Card Author and Contact

• —👩‍💻 Theresia Veronika Rampisela (thra[at]hum.ku.dk)