datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
parallel-sentences-ccmatrix
Dataset Card for Parallel Sentences - CCMatrix
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. The texts originate from the CCMatrix dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices
parallel-sentences-muse
parallel-sentences-jw300
parallel-sentences-news-commentary… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-ccmatrix.parallel-sentences-talks
Dataset Card for Parallel Sentences - Talks
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the Talks dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices
parallel-sentences-muse… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-talks.st-parallel-sentences
Dataset Card for "st-parallel-sentences"
More Information needed
parallel-sentences-opensubtitles
Dataset Card for Parallel Sentences - OpenSubtitles
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the OpenSubtitles dataset.
Warning! The quality of this dataset is not great; many of the english and non-english texts don't match well, or are fully empty.
Related Datasets
The following… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-opensubtitles.NCERT-Parallel-Dataset-Indicglobal-piqa-parallel
Global PIQA Parallel
Global PIQA is a participatory commonsense reasoning benchmark for over 100 languages, constructed by hand by over 350 researchers from over 65 countries around the world.
The parallel split is a multi-parallel dataset for 131 language varieties, covering five continents, 16 language families, and 23 writing systems.
In this parallel split, each example was machine-translated from English, then manually corrected by a native speaker of the target language.… See the full description on the dataset page: https://huggingface.co/datasets/mrlbenchmarks/global-piqa-parallel.parallel-sentences-tatoeba
Dataset Card for Parallel Sentences - Tatoeba
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the Tatoeba dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices
parallel-sentences-muse… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-tatoeba.bornholmsk_parallelThis dataset is parallel text for Bornholmsk and Danish.
For more details, see the paper [Bornholmsk Natural Language Processing: Resources and Tools](https://aclanthology.org/W19-6138/).parallel-sentences-wikimatrix
Dataset Card for Parallel Sentences - WikiMatrix
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the WikiMatrix dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-wikimatrix.kasem-speech-text-parallel
This dataset is shared under CC BY-NC 4.0, which means you are free to use, share, and adapt it for non-commercial research and educational purposes with attribution. You can read the full license at https://creativecommons.org/licenses/by-nc/4.0/.
Kasem Speech-Text Parallel Dataset
Dataset Description
This dataset contains 75990 parallel speech-text pairs for Kasem, a language spoken primarily in Ghana. The dataset consists of audio recordings paired with their… See the full description on the dataset page: https://huggingface.co/datasets/ghanaopenai/kasem-speech-text-parallel.parallel-sentences-jw300
Dataset Card for Parallel Sentences - JW300
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the JW300 dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices
parallel-sentences-muse… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-jw300.parallel-sentences-opus-100
Dataset Card for Parallel Sentences - OPUS-100
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. The sentences originate from the OPUS-100 website.
In particular, this dataset is a reformatting of the OPUS-100 dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-opus-100.robotwin_scan_object_place_dual_shoes_parallel_200parallel-sentences-europarl
Dataset Card for Parallel Sentences - Europarl
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the Europarl dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices
parallel-sentences-muse… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-europarl.polynews-parallel
Dataset Card for PolyNewsParallel
Dataset Summary
PolyNewsParallel is a multilingual paralllel dataset containing news titles for 833 language pairs. It covers 64 languages and 17 scripts.
Uses
This dataset can be used for machine translation or text retrieval.
Languages
There are 64 languages avaiable:
Code
Language
Script
amh_Ethi
Amharic
Ethiopic
arb_Arab
Modern Standard Arabic
Arabic
ayr_Latn
Central Aymara
Latin
bam_Latn
Bambara… See the full description on the dataset page: https://huggingface.co/datasets/aiana94/polynews-parallel.ga-speech-text-parallel-90k
This dataset is shared under CC BY-NC 4.0, which means you are free to use, share, and adapt it for non-commercial research and educational purposes with attribution. You can read the full license at https://creativecommons.org/licenses/by-nc/4.0/.
This dataset is made available because of Ghana NLP's volunteer driven research work. Please consider contributing to any of our projects on Github
Ga Speech-Text Parallel Dataset
Dataset Description
This dataset… See the full description on the dataset page: https://huggingface.co/datasets/ghanaopenai/ga-speech-text-parallel-90k.xone-repository-parallel-en-id-corpusWe are currently developing new version of LMSE translation scoring model and processing additional data sources. We estimate the dataset will expand, with significantly improved quality.(Delayed..)
A score of 55% and above indicates high-quality translation pairs, even if the first version of the model we developed gave them such a score. We will try to release a newer model in the future with better quality and consistently fast scoring speeds, and release it to the public once we decide… See the full description on the dataset page: https://huggingface.co/datasets/cloverx-id/xone-repository-parallel-en-id-corpus.GSM8KInstruct_Parallelvox-communis-parallel-g2p
VoxCommunis Parallel G2P dataset
This dataset was derived from the VoxCommunis Corpus to provide pairs of utterances along with their
corresponding phonemes, side by side, as to ease the training of grapheme-to-phoneme (G2P) models.
The original VoxCommunis Corpus features force-aligned TextGrids with phone- and word-level segmentations derived from the Mozilla Common Voice Corpus.
The lexicons were developed using Epitran, the XPF Corpus, Charsiu, and some custom dictionaries.… See the full description on the dataset page: https://huggingface.co/datasets/fdemelo/vox-communis-parallel-g2p.parallel-sentences-global-voices
Dataset Card for Parallel Sentences - Global Voices
This dataset contains parallel sentences (i.e. English sentence + the same sentences in another language) for numerous other languages. Most of the sentences originate from the OPUS website.
In particular, this dataset contains the Global Voices dataset.
Related Datasets
The following datasets are also a part of the Parallel Sentences collection:
parallel-sentences-europarl
parallel-sentences-global-voices… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences-global-voices.twi-trigrams-speech-text-parallel
This dataset is shared under CC BY-NC 4.0, which means you are free to use, share, and adapt it for non-commercial research and educational purposes with attribution. You can read the full license at https://creativecommons.org/licenses/by-nc/4.0/.
Twi Trigrams Speech-Text Parallel Dataset
Dataset Description
This dataset contains 166156 parallel speech-text pairs for Twi, a language spoken primarily in Ghana. The dataset consists of audio recordings of trigram… See the full description on the dataset page: https://huggingface.co/datasets/ghanaopenai/twi-trigrams-speech-text-parallel.Cyber-Parallel-Dataset-Indicgemstones_data_order_parallelGemstones Training Dataset - Parallel workers sharded version
This data is a reprocessed version of the first 1B rows of the Dolma v1.7 dataset (https://huggingface.co/datasets/allenai/dolma).
The data is encoded using the Pythia tokenizer: https://huggingface.co/EleutherAI/pythia-160m
Disclaimer: this is an approximation of the dataset used to train the Gemstones model suite.
Due to the randomized and sharded nature of the distributed training code, the only way to perfectly
reproduce the… See the full description on the dataset page: https://huggingface.co/datasets/tomg-group-umd/gemstones_data_order_parallel.parallel-sentences
Parallel Sentences for 50+ languages
[!NOTE]
This repository contains raw datasets, all of which have also been formatted for easy training in the Parallel Sentences Datasets collection. We recommend looking there first.
This repository contains parallel sentences (i.e. English + same sentences in other language) for 50+ different languages in a simple tsv.gz format:
english_sentences\tsentence_in_other_language
Sentences stem from the OPUS website.
The following datasets are… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/parallel-sentences.IISC_parallel_franka_all_fix_tracks_fixedThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "franka",
"total_episodes": 0,
"total_frames": 0,
"total_tasks": 0,
"total_videos": 0,
"total_chunks": 0,
"chunks_size": 1000,
"fps": 15,
"splits": {},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/hyzhang01/IISC_parallel_franka_all_fix_tracks_fixed.Finance-Parallel-Dataset-Indicrobotwin_put_obj_cabinet_parallel_50
ABORTED research artifact — retained for reproducibility, not deleted.
This artifact is retained as historical evidence only. Its cached segment-start main-camera observations are not eligible for dynamic-main-view claims. See ABORTED.yaml for the machine-readable archival record.
Archival registry mapping:
experiment_id: E002
This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "aloha"… See the full description on the dataset page: https://huggingface.co/datasets/Shiki42/robotwin_put_obj_cabinet_parallel_50.Parallel_Dataset
Dataset Sources
Paper: LegoMT2: Selective Asynchronous Sharded Data Parallel Training for Massive Neural Machine Translation
Link: https://aclanthology.org/2025.findings-acl.1200.pdf
Repository: https://github.com/CONE-MT/CONE
Computer-Science-Parallel-Dataset-IndicX-ALMA-Parallel-Data
This is the translation parallel dataset used by X-ALMA.
@misc{xu2024xalmaplugplay,
title={X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale},
author={Haoran Xu and Kenton Murray and Philipp Koehn and Hieu Hoang and Akiko Eriguchi and Huda Khayrallah},
year={2024},
eprint={2410.03115},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2410.03115},
}