eduagarcia/CrawlPT_dedup

CrawlPT (deduplicated)

CrawlPT is a generic Portuguese corpus extracted from various web pages.

This version is deduplicated using MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022). The raw version is also available here.

Dataset Details

Dataset is composed by three corpora: brWaC, C100-PT, OSCAR-2301.

• —brWaC: a web corpus for Brazilian Portuguese from 120,000 different websites.
• —C100-PT: Portuguese subset from CC-100. C100 was created for training the multilingual Transformer XLM-R, containing two terabytes of cleaned data from 2018 snapshots of the Common Crawl project in 100 languages. We use the , which contains 49.1 GiB of text.
• —OSCAR-2301-PT: curation from OSCAR-2301 in the Portuguese language.

Dataset Description

• —Language(s) (NLP): Brazilian Portuguese (pt-BR)
• —Repository: https://github.com/eduagarcia/roberta-legal-portuguese
• —Paper: https://aclanthology.org/2024.propor-1.38/

Data Collection and Processing

Raw corpora sizes in terms of billions of tokens and file size in GiB:

CrawlPT is deduplicated using MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022).

We used 5-grams and a signature of size 256, considering two documents to be identical if their Jaccard Similarity exceeded 0.7. Deduplicate rate found by the Minhash-LSH algorithm for the CrawlPT corpus:

Citation

@inproceedings{garcia-etal-2024-robertalexpt,
    title = "{R}o{BERT}a{L}ex{PT}: A Legal {R}o{BERT}a Model pretrained with deduplication for {P}ortuguese",
    author = "Garcia, Eduardo A. S.  and
      Silva, Nadia F. F.  and
      Siqueira, Felipe  and
      Albuquerque, Hidelberg O.  and
      Gomes, Juliana R. S.  and
      Souza, Ellen  and
      Lima, Eliomar A.",
    editor = "Gamallo, Pablo  and
      Claro, Daniela  and
      Teixeira, Ant{\'o}nio  and
      Real, Livy  and
      Garcia, Marcos  and
      Oliveira, Hugo Gon{\c{c}}alo  and
      Amaro, Raquel",
    booktitle = "Proceedings of the 16th International Conference on Computational Processing of Portuguese",
    month = mar,
    year = "2024",
    address = "Santiago de Compostela, Galicia/Spain",
    publisher = "Association for Computational Lingustics",
    url = "https://aclanthology.org/2024.propor-1.38",
    pages = "374--383",
}

Acknowledgment

This work has been supported by the AI Center of Excellence (Centro de Excelência em Inteligência Artificial – CEIA) of the Institute of Informatics at the Federal University of Goiás (INF-UFG).