leeaandrob/mirror-eduagarcia__LegalPT_dedup

LegalPT (deduplicated)

LegalPT aggregates the maximum amount of publicly available legal data in Portuguese, drawing from varied sources including legislation, jurisprudence, legal articles, and government documents.

This version is deduplicated using MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022). The raw version is also available here.

Dataset Details

Dataset is composed by six corpora: Ulysses-Tesemõ, MultiLegalPile (PT), ParlamentoPT, Iudicium Textum, Acordãos TCU, and DataSTF.

• —**MultiLegalPile** (Paper): a multilingual corpus of legal texts comprising 689 GiB of data, covering 24 languages in 17 jurisdictions. The corpus is separated by language, and the subset in Portuguese contains 92GiB of data, containing 13.76 billion words. This subset includes the jurisprudence of the Court of Justice of São Paulo (CJPG), appeals from the 5th Regional Federal Court (BRCAD-5), the Portuguese subset of legal documents from the European Union, known as EUR-Lex, and a filter for legal documents from MC4.
• —**Ulysses-Tesemõ**: a legal corpus in Brazilian Portuguese, composed of 2.2 million documents, totaling about 26GiB of text obtained from 96 different data sources. These sources encompass legal, legislative, academic papers, news, and related comments. The data was collected through web scraping of government websites.
• —**ParlamentoPT** (Paper): a corpus for training language models in European Portuguese. The data was collected from the Portuguese government portal and consists of 2.6 million documents of transcriptions of debates in the Portuguese Parliament.
• —**Iudicium Textum** (Paper): consists of rulings, votes, and reports from the Supreme Federal Court (STF) of Brazil, published between 2010 and 2018. The dataset contains 1GiB of data extracted from PDFs.
• —**Acordãos TCU** (Paper): an open dataset from the Tribunal de Contas da União (Brazilian Federal Court of Accounts), containing 600,000 documents obtained by web scraping government websites. The documents span from 1992 to 2019.
• —**DataSTF**): a dataset of monocratic decisions from the Superior Court of Justice (STJ) in Brazil, containing 700,000 documents (5GiB of data).

Dataset Description

• —Language(s) (NLP): Portuguese (pt-BR and pt-PT)
• —Repository: https://github.com/eduagarcia/roberta-legal-portuguese
• —Paper: https://aclanthology.org/2024.propor-1.38/

Data Collection and Processing

LegalPT is deduplicated using MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022).

We used 5-grams and a signature of size 256, considering two documents to be identical if their Jaccard Similarity exceeded 0.7.

Duplicate rate found by the Minhash-LSH algorithm for the LegalPT corpus:

Citation

@inproceedings{garcia-etal-2024-robertalexpt,
    title = "{R}o{BERT}a{L}ex{PT}: A Legal {R}o{BERT}a Model pretrained with deduplication for {P}ortuguese",
    author = "Garcia, Eduardo A. S.  and
      Silva, Nadia F. F.  and
      Siqueira, Felipe  and
      Albuquerque, Hidelberg O.  and
      Gomes, Juliana R. S.  and
      Souza, Ellen  and
      Lima, Eliomar A.",
    editor = "Gamallo, Pablo  and
      Claro, Daniela  and
      Teixeira, Ant{\'o}nio  and
      Real, Livy  and
      Garcia, Marcos  and
      Oliveira, Hugo Gon{\c{c}}alo  and
      Amaro, Raquel",
    booktitle = "Proceedings of the 16th International Conference on Computational Processing of Portuguese",
    month = mar,
    year = "2024",
    address = "Santiago de Compostela, Galicia/Spain",
    publisher = "Association for Computational Lingustics",
    url = "https://aclanthology.org/2024.propor-1.38",
    pages = "374--383",
}

Acknowledgment

This work has been supported by the AI Center of Excellence (Centro de Excelência em Inteligência Artificial – CEIA) of the Institute of Informatics at the Federal University of Goiás (INF-UFG).