BSC-LT/MULTI_corpus

Dataset Card for MULTI-Corpus

Dataset Description

• —Point of Contact: aiinstitutemt@bsc.es

Dataset Summary

This corpus was compiled as part of the TRAIN project (Traducción Automática para la Inclusión, Automatic Translation for Inclusion), funded by MCIN/AEI and ERDF. It aggregates 15,191,441 sentence-level entries covering four extremely low-resource languages: Tamazight/Amazigh (ZGH), Pashto (PS), Wolof (WO), and Romani (ROM), paired with one or more high-resource counterparts (English, Spanish, French), plus monolingual data for each low-resource language.

The dataset is intended primarily for training machine translation (MT) systems targeting under-resourced languages of migrant and minority communities in Spain. It consolidates 30 source datasets drawn from OPUS, the NLLB collection, CommonVoice, Tatoeba, Col·lectivaT, IRCAM, and other sources into a single unified large dataset (see Source Data section for the full list).

The data spans a wide range of domains including web-crawled text, software localisation, news, encyclopaedic content (Wikipedia), religious text (Bible), educational contents, and institutional documents.

Supported Tasks and Leaderboards

• —Machine Translation: parallel sentence pairs for the language pairs EN-ZGH, ES-ZGH, FR-ZGH, EN-PS, ES-PS, EN-WO, FR-WO, EN-ROM, ES-ROM.

Languages

Dataset Structure

This dataset is provided in two different versions:

The multi-corpus-raw.tsv file is the unfiltered aggregation of all 30 source datasets listed in the Source Data section. It contains only original (non-synthetic) data, covering original parallel pairs (14,493,051 rows) and monolingual sentences (698,390 rows) across the four low-resource languages ZGH, PS, WO, and ROM, paired with EN, ES, and FR. No filtering, deduplication, or synthetic data generation has been applied; the file preserves the data exactly as found in the source corpora. It is suited for exploratory analysis, custom filtering pipelines, and as the upstream source for the training set.

The multi-training-set.json file is a preprocessed, filtered, and augmented training-ready dataset derived from the raw corpus. It contains Spanish (ES) sentences paired with sentences in ZGH, PS, WO, and ROM, and was produced through three steps: (1) filtering the raw corpus with the language identification tool Fasttext, which significantly reduced the size of the noisier web-crawled datasets; (2) generating synthetic ES-XX parallel data by automatically translating monolingual low-resource sentences into Spanish using Google Translate; and (3) augmenting the ES-XX parallel data by pivot-translating existing EN-XX and FR-XX sentence pairs into Spanish.

Data Instances

The dataset multi-corpus-raw.tsv is provided in TSV (tab-separated values) format. It contains one sentence per row with the following structure:

{
  "type": "Parallel",
  "lang_low": "ZGH",
  "lang_high": "EN",
  "dataset": "commonvoice",
  "text_low": "ⴰⵔⴳⴰⵜ ⵙ ⵓⴳⵕⴹ ⵏⵏⵓⵏ",
  "text_high": "Contribute Your Voice"
}

Example monolingual row:

{
  "type": "Mono",
  "lang_low": "ROM",
  "lang_high": "-",
  "dataset": "curriculum-framework-for-romani",
  "text_low": "I Romani čhib si po drom te xasavol, te na avel la protekcija thaj ažutipe lovenca katar le nacionalni thaj internacionalni aktorura.",
  "text_high": "-"
}

Data Fields

• —type: Parallel (bilingual sentence pair) or Mono (monolingual sentence).
• —lang_low: ISO code of the low-resource language (ZGH, PS, WO, ROM).
• —lang_high: ISO code of the high-resource paired language (EN, ES, FR), or - for monolingual entries.
• —dataset: Name of the source dataset (see Source Data section for the full list).
• —text_low: Sentence in the low-resource language.
• —text_high: Paired sentence in the high-resource language, or - for monolingual entries.

The file multi-training-set.json is provided in JSON format. The structure follows the input format required by MADLAD-400, the multilingual MT model used for experimentation in the TRAIN project. Specifically, the target low-resource language is indicated by a language tag of the form <2xx> prepended to the Spanish source sentence in the src field, while the tgt field holds the corresponding sentence in the low-resource language:

{
  "src": "<2wo> El Océano Austral es el cuerpo de agua que rodea el Océano Austral.",
  "tgt": "Mbàmbulaan gu Bëj-saalum gi mooy mbalkaam ndox mi ne ci li wër Dottub Bëj-saalum bi."
}

Data Splits

The dataset contains a single split: train.

Raw Dataset Composition

Parallel rows: 14,493,051 - Monolingual rows: 698,390

Dataset Creation

Curation Rationale

The TRAIN project (Task 2.4 Collection of textual resources for the targeted languages) aimed to compile at least 300,000 sentences per language to support the development of MT systems for the migrant and minority languages of communities in Spain. Due to the extreme scarcity of bilingual corpora for these languages, the collection strategy targeted two complementary types of resources:

1. 1.Bilingual corpora: the preferred type, covering any language pair involving the low-resource language and at least one high-resource partner (Spanish, English, or French as pivot).
2. 2.Monolingual corpora: used to supplement bilingual resources and to generate synthetic parallel data via machine translation and backtranslation.

And additionally applied:

1. 1.Synthetic data generation strategies included pivot-language translation (e.g., Pashto-English → Spanish) and backtranslation from monolingual data. Synthetic data are included exclusively in the multi-training-set.json file.

Source Data

Initial Data Collection and Normalization

Bilingual source datasets:

Monolingual source datasets:

Data Filtering and Normalization:

• —The source parallel datasets from OPUS and NLLB had already been filtered with language identification Fasttext prior to inclusion in this corpus.
• —Wikipedia dumps were cleaned and split into sentences before inclusion.
• —The Corpus IRCAM originally contains text in both Latin and Tifinagh scripts; the texts written in the Latin script were transliterated into Tifinagh by using a Tifinagh transliterator.
• —No additional filtering or deduplication has been applied during the aggregation step that produced this corpus.
• —An additional language identification pass with Fasttext was applied to the multi-training-set.json dataset to remove sentence pairs where the low-resource or high-resource language was misidentified. This significantly reduced the size of the noisier web-crawled datasets. Monolingual low-resource sentences that passed the filter were machine-translated into Spanish with Google Translate to generate synthetic parallel data. Additionally, existing EN-XX and FR-XX parallel pairs were automatically translated into Spanish via pivot translation to augment the ES-XX training data, given the project's primary goal of building Spanish↔low-resource MT systems.

Who are the source language producers?

• —Tamazight: Amazigh-speaking communities; data curated by Col·lectivaT (with funding from the Municipality of Barcelona and the Government of Catalonia), IRCAM (Institut Royal de la Culture Amazighe / TALAM), and Wikipedia contributors.
• —Pashto: Pashto-speaking communities in Afghanistan, Pakistan, and diaspora; data from OPUS/NLLB (OPUS) and Wikipedia contributors.
• —Wolof: Wolof-speaking communities in Senegal, The Gambia, and Mauritania; data from OPUS/NLLB (OPUS) and Wikipedia contributors.
• —Romani: Romani-speaking communities across Europe; data from OPUS/Bible-UEdin, Wikipedia contributors, and the Council of Europe (institutional documents).

Annotations

Annotation process

The dataset does not contain any manual annotations beyond the parallel alignments, which were either preserved from source datasets or validated through automated alignment scoring.

Who are the annotators?

[N/A]

Personal and Sensitive Information

Given that this dataset is derived from pre-existing datasets that contain crawled data, and that no specific anonymisation process has been applied, personal and sensitive information may be present in the data. This needs to be considered when using the data for training models.

Considerations for Using the Data

Social Impact of Dataset

This dataset contributes resources for four extremely under-resourced languages spoken by migrant and minority communities. Improved MT systems for these languages can facilitate communication with public services, support access to information, and help preserve and document these languages digitally. This is especially relevant in the context of Spain, where Romani, Tamazight-speaking Amazigh communities, and migrant communities speaking Pashto and Wolof are present.

Discussion of Biases

No specific bias mitigation strategies were applied to this dataset beyond deduplication and minimal quality filtering. Inherent biases may exist within the data, reflecting the biases present in the source datasets, which include web-crawled content, subtitles, news articles, and other user-generated or institutionally produced text.

Additional content-specific considerations:

• —Domain imbalance: Pashto data is heavily dominated by the NLLB corpus (11.3M of 12.7M total PS rows), which is web-crawled. Other domains are relatively small.
• —Dialectal variation: Romani in particular has significant dialectal diversity; the data does not represent all dialects equally. Tamazight similarly covers primarily Standard Moroccan Amazigh (ZGH) rather than the full Amazigh dialect spectrum.
• —Script inconsistencies: Tamazight data may contain mixed Tifinagh/Latin representations in some source files. Additionally, the mentioned Tifinagh transliterator is an experimental tool whose results have not been verified for accuracy or quality.

Other Known Limitations

• —Reduced domain coverage for Romani (primarily Bible text and Wikipedia) and for some Tamazight pairs.
• —Known limitations of crawled data apply: noise, misalignments, and low-quality sentence pairs may be present, particularly in large web-crawled corpora (NLLB, CCAligned, MultiParaCrawl).
• —The synthetic parallel data in multi-training-set.json was generated by machine translation (Google Translate) and is therefore subject to the inherent limitations of MT systems, including translation errors, hallucinations, and propagation of the source model's biases. Synthetic sentences may also lack the naturalness and variability of authentic text, which can negatively affect the fluency of MT models trained on them.

Additional Information

Dataset Curators

Machine Translation Group, AI Institute, Barcelona Supercomputing Center (aiinstitutemt@bsc.es).

Funding

This work has been supported by the Spanish project PID2021-123988OB-C33 funded by MCIN/AEI/10.13039/501100011033/FEDER, UE, as part of the TRAIN project (Traducción Automática para la Inclusión), Work Package 2, Task 2.4.

Acknowledgements

The Tamazight data was collected with support of Col·lectivaT. The Tamazight monolingual corpus was provided by IRCAM trough the portal TALAM (Traitement Automatique de la Langue Amazighe). Several datasets were obtained from OPUS, an open collection of multilingual parallel corpora (Tiedemann, J., 2012. Parallel Data, Tools and Interfaces in OPUS. Proceedings of LREC 2012). Each OPUS dataset retains the licence of its original source; credit is due to the respective dataset creators and contributors.

Licensing Information

This work is licensed under a Creative Commons Attribution ShareAlike 4.0 International licence.

Note: Individual source datasets carry their own licences. Users of this aggregated corpus should verify licence compatibility with the intended use case.

Users must ensure compliance with all applicable third-party licenses.

Citation Information

[N/A]

Contributions

[N/A]