CoolFace
Datasetpublic

wecover/OPUS

Collection of OPUS Corpus from https://opus.nlpl.eu has been collected. The following corpora have been included: UNPC GlobalVoices TED2020 News-Commentary WikiMatrix Tatoeba Europarl OpenSubtitles 25,000 samples (randomly sampled within the first 100,000 samples) per language pair of each corpus were collected, with no modification of data. Licenses OPUS @inproceedings{tiedemann2012parallel, title={Parallel data, tools and interfaces in OPUS.}… See the full description on the dataset page: https://huggingface.co/datasets/wecover/OPUS.

sourceHugging Faceupdated 2y agoView on Hugging Face
0likes5.2kdownloads
Dataset Card

Collection of OPUS

Corpus from https://opus.nlpl.eu has been collected. The following corpora have been included:

25,000 samples (randomly sampled within the first 100,000 samples) per language pair of each corpus were collected, with no modification of data.

Licenses

OPUS

@inproceedings{tiedemann2012parallel,
  title={Parallel data, tools and interfaces in OPUS.},
  author={Tiedemann, J{\"o}rg},
  booktitle={Lrec},
  volume={2012},
  pages={2214--2218},
  year={2012},
  organization={Citeseer}
}

Tatoeba

CC BY 2.0 FR

TED2020

CC BY–NC–ND 4.0

@inproceedings{reimers-2020-multilingual-sentence-bert,
    title = "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2020",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2004.09813",
}

WikiMatrix

CC-BY-SA 4.0
@article{schwenk2019wikimatrix,
  title={Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia},
  author={Schwenk, Holger and Chaudhary, Vishrav and Sun, Shuo and Gong, Hongyu and Guzm{\'a}n, Francisco},
  journal={arXiv preprint arXiv:1907.05791},
  year={2019}
}

UNPC

@inproceedings{ziemski2016united,
  title={The united nations parallel corpus v1. 0},
  author={Ziemski, Micha{\l} and Junczys-Dowmunt, Marcin and Pouliquen, Bruno},
  booktitle={Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  pages={3530--3534},
  year={2016}
}