tomasmajercik/flores-parquet

FLORES Parquet

Parquet version of the FLORES dataset for efficient streaming.

⚠️ This is a derivative work: This dataset is a reformatted version of the original FLORES dataset created by Meta AI. All credit goes to the original authors. This version simply converts the data to Parquet format for easier streaming and usage.

Usage

from datasets import load_dataset

# Load specific language
ds = load_dataset("tomasmajercik/flores-parquet", name="fra_Latn", split="validation")
print(ds[0])

Dataset Information

This dataset is derived from the FLORES benchmark, which is designed for evaluating machine translation systems across a diverse set of languages.

Each language subset contains parallel sentences that can be used for:

• —Machine translation evaluation
• —Cross-lingual model testing
• —Multilingual NLP research

Changes from Original

This version differs from the original FLORES dataset only in format:

• —Converted from the original format to Parquet for efficient streaming
• —Organized into language-specific configs for easy access
• —No changes to the actual content or data

License

This dataset is licensed under CC-BY-SA-4.0 (Creative Commons Attribution-ShareAlike 4.0 International), the same license as the original FLORES dataset.

License Terms

You are free to:

1. 1.Share — copy and redistribute the material in any medium or format for any purpose, even commercially.
2. 2.Adapt — remix, transform, and build upon the material for any purpose, even commercially.
3. 3.The licensor cannot revoke these freedoms as long as you follow the license terms.

Under the following terms:

1. 1.Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
2. 2.ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
3. 3.No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

Full license text: https://creativecommons.org/licenses/by-sa/4.0/

Citation

If you use this dataset, please cite the original FLORES papers:

Primary Citation (NLLB / FLORES-200)

@article{nllb2022,
  author    = {NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi,  Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Jeff Wang},
  title     = {No Language Left Behind: Scaling Human-Centered Machine Translation},
  year      = {2022}
}

FLORES-101

@inproceedings{,
  title={The FLORES-101  Evaluation Benchmark for Low-Resource and Multilingual Machine Translation},
  author={Goyal, Naman and Gao, Cynthia and Chaudhary, Vishrav and Chen, Peng-Jen and Wenzek, Guillaume and Ju, Da and Krishnan, Sanjana and Ranzato, Marc'Aurelio and Guzm'{a}n, Francisco and Fan, Angela},
  year={2021}
}

Original FLORES

@inproceedings{,
  title={Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English},
  author={Guzm'{a}n, Francisco and Chen, Peng-Jen and Ott, Myle and Pino, Juan and Lample, Guillaume and Koehn, Philipp and Chaudhary, Vishrav and Ranzato, Marc'Aurelio},
  journal={arXiv preprint arXiv:1902.01382},
  year={2019}
}

Acknowledgments

This dataset is a derivative work based on the FLORES dataset created by Meta AI Research. All credit for the original data collection, curation, and annotation goes to the original FLORES team and contributors.

Links

• —Original FLORES Dataset: facebook/flores
• —FLORES GitHub: https://github.com/facebookresearch/flores
• —License: CC-BY-SA-4.0

Available Languages