sentence-transformers/parallel-sentences-opensubtitles

Dataset Card for Parallel Sentences - OpenSubtitles

This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website. In particular, this dataset contains the OpenSubtitles dataset.

Warning! The quality of this dataset is not great; many of the english and non-english texts don't match well, or are fully empty.

Related Datasets

The following datasets are also a part of the Parallel Sentences collection:

• —parallel-sentences-europarl
• —parallel-sentences-global-voices
• —parallel-sentences-muse
• —parallel-sentences-jw300
• —parallel-sentences-news-commentary
• —parallel-sentences-opensubtitles
• —parallel-sentences-talks
• —parallel-sentences-tatoeba
• —parallel-sentences-wikimatrix
• —parallel-sentences-wikititles
• —parallel-sentences-ccmatrix

These datasets can be used to train multilingual sentence embedding models. For more information, see sbert.net - Multilingual Models.

Dataset Subsets

all subset

• —Columns: "english", "non_english"
• —Column types: str, str
• —Examples:

    {
      "english": "We can't predict it and we can't control it.",
      "non_english": "نحن لا نَستطيعُ تَوَقُّعه ونحن لا نَستطيعُ السَيْطَرَة عليه."
    }

• —Collection strategy: Combining all other subsets from this dataset.
• —Deduplified: No

en-... subsets

• —Columns: "english", "non_english"
• —Column types: str, str
• —Examples:

    {
      "english": "ever faithful, ever true, nothing stops him, he'll get through.",
      "non_english": "우리의 한결같은 심부름꾼 황새 아저씨 가는 길을 그 누가 막으랴!"
    }

• —Collection strategy: Processing the raw data from parallel-sentences and formatting it in Parquet, followed by deduplication.
• —Deduplified: Yes