Datasetpublic

sentence-transformers/parallel-sentences

Parallel Sentences for 50+ languages [!NOTE] This repository contains raw datasets, all of which have also been formatted for easy training in the Parallel Sentences Datasets collection. We recommend looking there first. This repository contains parallel sentences (i.e. English + same sentences in other language) for 50+ different languages in a simple tsv.gz format: english_sentences\tsentence_in_other_language Sentences stem from the OPUS website. The following datasets are… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences.

sourceHugging Faceupdated 2y agoView on Hugging Face

15likes849downloads

Dataset Card

Parallel Sentences for 50+ languages

[!NOTE] This repository contains raw datasets, all of which have also been formatted for easy training in the Parallel Sentences Datasets collection. We recommend looking there first.

This repository contains parallel sentences (i.e. English + same sentences in other language) for 50+ different languages in a simple tsv.gz format:

english_sentences\tsentence_in_other_language

Sentences stem from the OPUS website.

The following datasets are included:

—Europarl
—GlobalVoices
—JW300
—MUSE
—News-Commentary
—OpenSubtitles
—Tatoeba
—Talks - Custom translated transcripts of talks
—WikiMatrix
—WikiTitles - Custom dataset with parallel Wikipedia titles

Usage

These sentences can be used to train multi-lingual sentence embedding models. For more details, see SBERT.net - Multilingual-Model

This dataset can not yet be used with Hugging Face dataset library. You must download the individual TSV files.