projecte-aina/CATalog

Dataset Description

• —Homepage Projecte AINA
• —Repository HuggingFace
• —Paper ["A CURATEd CATalog: Rethinking the Extraction of Pretraining Corpora for Mid-Resourced Languages"]()
• —Leaderboard N/A
• —Point of Contact langtech@bsc.es

Dataset Summary

CATalog is a diverse, open-source Catalan corpus for language modelling. It consists of text documents from 26 different sources, including web crawling, news, forums, digital libraries and public institutions, totaling in 17.45 billion words.

Supported Tasks and Leaderboards

• —Fill-Mask
• —Text Generation
• —other:Language-Modelling: The dataset is suitable for training a model in Language Modelling, predicting the next word in a given context. Success is measured by achieving a low Perplexity score, indicating the model's proficiency in accurately predicting subsequent words.
• —other:Masked-Language-Modelling: The dataset is designed for training models in Masked Language Modelling. This task involves predicting masked or hidden words within a sentence. Success is typically measured by achieving a high performance score, such as accuracy or F1 score, on correctly predicting the masked tokens.

Languages

This dataset is in Catalan (ca-ES). Coming from the web, some documents may contain other languages.

Dataset Structure

Data Instances

The dataset is provided in a JSONL format, where each row corresponds to a single document and contains a document identifier, the text, a quality score, the strategy used to evaluate the document quality, languages, and a URL of the document, if available.

{
  "id": "macocu_ca_20230731_9_402472",
  "text": "Jaume Casañas relleva Dolors Carreras a l’Alcaldia de l’Ajuntament de Cunit.
          La substitució prevista al pacte de govern del 2019 s’ha materialitzat aquest
          dissabte al matí. Aquest dissabte al matí, en un acte al Casal Municipal de
          Cunit, s’ha celebrat l’acte de relleu de l’Alcaldia de l’Ajuntament de Cunit,
          segons preveia el pacte de govern signat el juny del 2019 pels grups del PSC,
          encapçalat per la fins ara alcaldessa, Dolors Carreras, i Impulsem Cunit, amb
          el ja nou alcalde, Jaume Casañas, al capdavant.",
  "score": 0.8105327621841463,
  "strategy": "curate",
  "languages": "{"ca": 1.0}",
  "url": ""
}

Data Fields

• —id: text string containing the document identifier. Consists of the subdataset code, the part number and a document number.
• —text: text string from the document, with paragraphs separated by two newlines escape sequences. It is meant to be used directly as input for language modelling.
• —score: positive float number representing the document quality, ranging from 0, which represents the worst quality, to 1, the best quality.
• —strategy: text string describing the type of evaluation applied to obtain the document score. "curate" uses the heuristic evaluation from CURATE and "perfect" means that manual review was done and the highest score (1) is applied.
• —languages: dictionary containing the document languages, with a percentage indicating the character ratio for each one.
• —url: text string with the URL of the document, if available.

Data Splits

We do not provide any canonical splits for CATalog.

Dataset Creation

Curation Rationale

CATalog is mainly built on filtered, non-overlapping versions of CommonCrawl snapshots and a smaller set of manually selected corpora from specific sources. We use the CURATE pipeline, which combines exact deduplication, language identification, and scoring heuristics.

In the design of CATalog, we adhere to the following values:

• —(1) Scale & Flexibility. We intend to produce datasets that have a significant impact on the training of multilingual models in the range of 7B-180B parameters. Since Catalan is a medium-resource language and data acquisition is already a challenge, binary filtering will limit us in terms of the amount of data. By providing a score, we are able to easily filter the corpus according to any requirement.
• —(2) Neutral scoring. As opposed to ML-based filtering, we use simple rules and heuristics to avoid introducing further bias into the model (Dodge et al., 2021; Welbl et al., 2021). We only use FastText to reject documents in other languages.

During development, we performed comparative judgment experiments to evaluate the usefulness of the scoring from the CURATE pipeline, which is intended for further filtering and analysis. We found a moderate correlation between the score and the perceived quality of the text. Our main goal was to maximize the usability of the corpus without getting into a trade-off between quantity and quality.

Source Data

Initial Data Collection and Normalization

We applied extensive data processing using our CURATE pipeline.

We first filter documents by their language content using FastText. Only documents with at least 50% of characters in Catalan are kept. We then perform exact document deduplication. After this stage, we score each document with a tested set of 8 heuristic evaluators, inspired from other web filterings and from our own creation.

The following pre-existing datasets were used:

• —`OSCAR-2301`
• —`OSCAR-2201`
• —`CaText`
• —`MaCoCu-ca 1.0`
• —`caWaC`
• —`Colossal OSCAR 1.0`
• —`mC4`

Who are the source language producers?

Apart from the pre-existing datasets, all of them coming from CommonCrawl dumps, the following sources provided their data on Open Data Agreements:

• —## Media Groups
• —`IB3`
• —`Grup El Món`
• —`Vilaweb`
• —`Nació Digital`
• —`ACN`
• —`Racó Català Articles`
• —`Racó Català Fòrums (anonymized version)`
• —`Aquí Berguedà`
• —## Academic & Book Repositories
• —`Tesis Doctorals en Xarxa (TDX)`
• —`Wikipedia`
• —`Project Gutenberg`
• —## Government Institutions
• —`Parlament de Catalunya`
• —`Les Corts Valencianes`
• —`Diari Oficial de la Generalitat Valenciana`
• —`Butlletí Oficial de la Universitat d'Alacant`

Annotations

The score is an automatic label obtained from the aggregation of different heuristic evaluators based on predefined thresholds. Specific evaluators penalize documents for factors like minimum word count, average word per sentence, punctuation per word rate, unique sentences ratio, stopword ratio, Brunet index, language diversity, and content identified by regular expressions, providing a comprehensive approach to document scoring.

Annotation process

The process involves assigning scores between 0 and 1 to sentences, paragraphs, and documents in a hierarchical manner. Individual evaluators at different levels contribute scores that are combined using geometric means, emphasizing a probability-like interpretation to encourage evaluators to assess desirability. The final document score is derived through analogous aggregation of paragraph and document scores, distinct from a linear model.

Who are the annotators?

[N/A]

Personal and Sensitive Information

Being partially constructed from Common Crawl, personal and sensitive information might be present. This must be considered before training deep learning models with CATalog, specially in the case of text-generation models.

Considerations for Using the Data

Social Impact of Dataset

CATalog promotes the Catalan language in the NLP field, enabling development of advanced applications and chatbots tailored to Catalan speakers, while improving access to information for better community understanding. However, most of the sources in the dataset are web-scraped, which may bring in biases and privacy issues, risking flawed outcomes and potential misuse.

Given that Catalan is a mid-resourced language with low representation in digital sources, this dataset becomes crucial for building inclusive NLP applications. It addresses the language's underrepresentation, empowering the Catalan community with improved access to text resources in their native language. However, careful consideration of potential biases and privacy issues is essential to ensure responsible and equitable technology use.

Discussion of Biases

Web-crawled content is over-represented with standard language varieties, impacting language model performance for minority languages. Language diversity in data is crucial to avoid bias, especially in encoding non-standard dialects, preventing the exclusion of demographic groups. Our corpus primarily focuses on Central Catalan, but we actively include Valencian and Balearic Catalan, along with diverse sociolects from platforms like Racó Català Fòrums, aiming for a more representative dataset. Despite legal uncertainties in web-scraped data, we prioritize permissive licenses and privacy protection measures, acknowledging the challenges posed by personally identifiable information (PII) within large-scale datasets. Our ongoing efforts aim to address privacy concerns and contribute to a more inclusive linguistic dataset.

Other Known Limitations

[N/A]

Additional Information

Dataset Curators

Language Technologies Unit (langtech@bsc.es) at the Barcelona Supercomputing Center (BSC).

Funding

This work has been promoted and financed by the Generalitat de Catalunya through the Aina project and by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project ILENIA with reference 2022/TL22/00215337

Licensing Information

Licensing and Usage Terms: CATalog is a curated aggregation of multiple data sources compiled by BSC-CNS for research and development in natural language processing. The dataset as a whole does not carry a unified license. The use, reuse, and redistribution of each part of this dataset is governed by the license of its original source. You can find a detailed list of all sources and their associated licenses in the Licensing Information section and in the licenses.json file. Please consult the license of each dataset before using it in your projects. If you intend to redistribute this dataset, you must include the licensing metadata (e.g., licenses.json) alongside the data as a clear reporting of its mixed licensing structure. There is no explicit data intermediary mechanism; BSC-CNS has collected and used the data according to the terms of each data source license or through private Data Sharing Agreements. BSC-CNS acts solely as a curator and aggregator and does not assert ownership or control over third-party content included in this dataset. Users are solely responsible for ensuring compliance with the respective licenses and terms of use for each data source.

Disclaimer: CATalog is provided "as is", without warranty of any kind, express or implied, including but not limited to warranties of merchantability or fitness for a particular purpose. The dataset has been developed at a research institution (BSC-CNS) as part of ongoing experimental research and is therefore subject to errors, omissions, and inaccuracies. This dataset is not a product and therefore has not been subjected to quality control. BSC-CNS makes no guarantees regarding the accuracy, completeness, non-infringement, reliability, or suitability or of the dataset for any specific purpose. Use of CATalog is at the user’s own risk.

The list of SPDX license identifiers included in the documentation can be found in the following table or in this JSON file.

\* The data from each entity is governed by a distinct Data Sharing Agreement. All data provided by these entities is open and freely distributable.

Citation Information

[N/A]

Contributions

We thank the VIVES Plan for language technologies of the Valencian community, https://vives.gplsi.es/, from the CENID Digital Intelligence Center of the University of Alicante and the DFKI for their collaboration and contribution.