CoolFace
Datasetpublic

Arshia82sbn/Translation-Dataset-Large

Translation-Dataset_Large 🌍 A massive, unified multilingual parallel corpus for Persian, English, and Arabic NMT research. Translation-Dataset_Large aggregates and normalizes millions of sentence pairs across three language directions — English↔Persian (en-fa), Arabic↔English (ar-en), and Arabic↔Persian (ar-fa) — into a single, deduplicated, research-ready .parquet dataset. Dataset Summary Property Value Languages Persian (fa), English (en), Arabic… See the full description on the dataset page: https://huggingface.co/datasets/Arshia82sbn/Translation-Dataset-Large.

sourceHugging Facecc-by-nc-4.0updated 2mo agoView on Hugging Face
0likes287downloads
Dataset Card

Translation-Dataset_Large 🌍

A massive, unified multilingual parallel corpus for Persian, English, and Arabic NMT research.

Translation-Dataset_Large aggregates and normalizes millions of sentence pairs across three language directions — English↔Persian (`en-fa`), Arabic↔English (`ar-en`), and Arabic↔Persian (`ar-fa`) — into a single, deduplicated, research-ready .parquet dataset.

Dataset Summary

PropertyValue
LanguagesPersian (fa), English (en), Arabic (ar)
Language Pairsen-fa, ar-en, ar-fa
FormatApache Parquet
LicenseCC-BY-NC-4.0
Size1M–10M sentence pairs
TaskNeural Machine Translation (NMT), Seq2Seq

Why Use This Dataset?

  • —Largest open Persian–Arabic–English parallel corpus on Hugging Face
  • —Suitable for fine-tuning mT5, NLLB, MarianMT, mBART, and Helsinki-NLP models
  • —Covers literary, news, subtitle, religious, and web domains for robust generalization
  • —Fully deduplicated and normalized — no preprocessing required
  • —Compatible with the 🤗 datasets library for one-line loading

Dataset Structure

Each row represents a single parallel translation pair:

ColumnTypeDescription
pairstringLanguage direction (e.g., en-fa, ar-en, ar-fa)
corpusstringSource corpus (e.g., MIZAN, TEP, FLORES-200, OpenSubtitles)
source_langstringBCP-47 language code of the source text
target_langstringBCP-47 language code of the target text
source_textstringOriginal sentence
target_textstringTranslated sentence

🛠️ How to Use

python
from datasets import load_dataset

dataset = load_dataset("Arshia82sbn/Translation-Dataset_Large")
print(dataset["train"][0])

Filter by language pair:

python
fa_en = dataset["train"].filter(lambda x: x["pair"] == "en-fa")

---

## Data Collection & Processing Pipeline

This dataset was programmatically aggregated from multiple open-source corpora using an automated pipeline:

1. **Format Normalization** — Converted OPUS Moses-format files (`.src`/`.tgt`), raw text files, and Hugging Face dictionaries into a unified schema.
2. **Alignment & Cleaning** — Paired lines positionally, truncated mismatched files to the shorter length, and dropped empty or `NaN` translation strings.
3. **Deduplication** — Removed exact duplicate pairs based on `(corpus, pair, source_text, target_text)`.
4. **Efficient Export** — Saved as chunked `.parquet` files for fast streaming and loading.

---

## Source Corpora

| Corpus | Pairs | Domain |
|---|---|---|
| MIZAN | en-fa | Literary (~1M sentences) |
| TEP | en-fa | General |
| OpenSubtitles | en-fa, ar-en | Subtitles |
| WikiMatrix | en-fa, ar-en | Wikipedia |
| TED2020 | en-fa, ar-en | Talks |
| Tatoeba | en-fa, ar-en | Community |
| Tanzil | ar-fa, ar-en | Religious (Quran) |
| FLORES-200 | en-fa, ar-en, ar-fa | Evaluation benchmark |
| OPUS-100 | en-fa, ar-en | Multilingual |
| CCAligned / CCMatrix | en-fa, ar-en | Web-crawled |
| GNOME / KDE4 / Ubuntu | en-fa | Software UI |
| UNPC | ar-en | UN documents |
| JW300 | ar-en, ar-fa | Religious |

---

## ⚖️ Attribution & Copyright

This dataset is a derivative work. All credit for original data collection belongs to the respective authors. **You must cite the original sources if you use this dataset.**

**MIZAN**
> Kashefi, O. *Mizan: A large Persian-English parallel corpus.* [GitHub](https://github.com/omidkashefi/Mizan)

**OPUS Corpora** (TEP, OpenSubtitles, WikiMatrix, TED2020, Tatoeba, Tanzil, GNOME, KDE4, Ubuntu, CCAligned, CCMatrix, UNPC, JW300)
> Tiedemann, J. (2012). Parallel Data, Tools and Interfaces in OPUS. *Proceedings of LREC'12*.

**FLORES-200 & NLLB (Meta AI)**
> NLLB Team et al. (2022). *No Language Left Behind: Scaling Human-Centered Machine Translation.* arXiv:2207.04672.

**OPUS-100**
> Zhang, B. et al. (2020). *Improving Massively Multilingual NMT and Zero-Shot Translation.* ACL 2020.

---

## License

Distributed under [`CC-BY-NC-4.0`](https://creativecommons.org/licenses/by-nc/4.0/) to respect the non-commercial and attribution constraints of the underlying source datasets. **Commercial use is not permitted.**

---

## Citation

If you use this dataset, please cite the original corpora listed above and link back to this repository.