datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
xone-repository-parallel-en-id-corpusWe are currently developing new version of LMSE translation scoring model and processing additional data sources. We estimate the dataset will expand, with significantly improved quality.(Delayed..)
A score of 55% and above indicates high-quality translation pairs, even if the first version of the model we developed gave them such a score. We will try to release a newer model in the future with better quality and consistently fast scoring speeds, and release it to the public once we decide… See the full description on the dataset page: https://huggingface.co/datasets/cloverx-id/xone-repository-parallel-en-id-corpus.ParallelKernelBench_Problems
ParallelKernelBench (benchmark)
Reference problems for ParallelKernelBench: a benchmark for LLM-generated multi-GPU CUDA kernels.
This dataset contains 87 reference implementations in reference/ and the input tensor specification in utils/input_output_tensors.py. Inputs are deterministic — reproduce them with create_input_tensor(rank, world_size, problem_id, base_shape, dtype, trial) from that file; you do not need stored .pt files.
Files
Path
Description… See the full description on the dataset page: https://huggingface.co/datasets/togethercomputer/ParallelKernelBench_Problems.ParallelKernelBench_Problems
ParallelKernelBench (benchmark)
Reference problems for ParallelKernelBench: a benchmark for LLM-generated multi-GPU CUDA kernels.
This dataset contains 87 reference implementations in reference/ and the input tensor specification in utils/input_output_tensors.py.
Files
Path
Description
data/problems.parquet
One row per problem (tabular access)
reference/*.py
Reference solution() implementations
utils/input_output_tensors.py
Input/output tensor… See the full description on the dataset page: https://huggingface.co/datasets/willychan21/ParallelKernelBench_Problems.bashkir-wikipedia-parallel
Bashkir-Russian Wikipedia Parallel Corpus
Sentence-level Bashkir-Russian parallel text from Wikipedia, scored and filtered
for machine translation.
Overview
Sentence-level Bashkir-Russian parallel dataset extracted from the corresponding
Bashkir and Russian Wikipedia dumps dated 2026-08-01. Candidate pairs are scored
for semantic alignment with multilingual sentence encoders (Meta LASER3, Google
LaBSE) and the in-domain Bashkir-Russian Pair Scorer. The filtered… See the full description on the dataset page: https://huggingface.co/datasets/failed09/bashkir-wikipedia-parallel.open_parallel_think_code_source
open_parallel_think_code_source
A large-scale code reasoning distillation dataset with 320,000 solution trajectories generated by 4 state-of-the-art thinking models across 10,000 unique coding problems.
Source / raw pool. This is the per-trajectory dataset. The packed parallel-thinking datasets derived from it are haowu89/open_parallel_think_code_full (full reasoning + solution) and haowu89/open_parallel_think_code_cot (solution only). Each trajectory's metadata carries… See the full description on the dataset page: https://huggingface.co/datasets/haowu89/open_parallel_think_code_source.human-ai-parallel-corpus-biber
Citation
If you use the corpus as part of your research, please cite:
Do LLMs write like humans? Variation in grammatical and rhetorical styles
@misc{reinhart2024llmswritelikehumans,
title={Do LLMs write like humans? Variation in grammatical and rhetorical styles},
author={Alex Reinhart and David West Brown and Ben Markey and Michael Laudenbach and Kachatad Pantusen and Ronald Yurko and Gordon Weinberg},
year={2024},
eprint={2410.16107}… See the full description on the dataset page: https://huggingface.co/datasets/browndw/human-ai-parallel-corpus-biber.coca-ai-parallel-corpus-biber
COCA-AI Parallel Corpus (Biber Parsed)
R users can import the data directly using r-polars:
library(polars)
df <- pl$read_parquet('hf://datasets/browndw/coca-ai-parallel-corpus-biber/**/*.parquet')
df <- df$to_data_frame()
Citation
If you use the corpus as part of your research, please cite:
Do LLMs write like humans? Variation in grammatical and rhetorical styles@misc{reinhart2024llmswritelikehumans,
title={Do LLMs write like humans? Variation in grammatical… See the full description on the dataset page: https://huggingface.co/datasets/browndw/coca-ai-parallel-corpus-biber.tunisian-msa-parallel-corpus
Dataset Description
This is an ambitious project to create a high-quality, reproducible parallel corpus for Modern Standard Arabic (MSA) and Tunisian Arabic (aeb) through a sophisticated synthetic data generation pipeline. The dataset is being developed by the Tunisia.AI community to address the scarcity of high-quality dialectal data for training and evaluating language models.
The primary goal is to provide a rich, well-documented resource for the research and development of:… See the full description on the dataset page: https://huggingface.co/datasets/tunis-ai/tunisian-msa-parallel-corpus.human-ai-parallel-corpus-spacy
Citation
If you use the corpus as part of your research, please cite:
Do LLMs write like humans? Variation in grammatical and rhetorical styles
@misc{reinhart2024llmswritelikehumans,
title={Do LLMs write like humans? Variation in grammatical and rhetorical styles},
author={Alex Reinhart and David West Brown and Ben Markey and Michael Laudenbach and Kachatad Pantusen and Ronald Yurko and Gordon Weinberg},
year={2024},
eprint={2410.16107}… See the full description on the dataset page: https://huggingface.co/datasets/browndw/human-ai-parallel-corpus-spacy.human-ai-parallel-corpus-docuscope
COCA-AI Parallel Corpus (Biber Parsed)
Data were tagged with the en_docusco_spacy model.
R users can import the data directly using r-polars:
library(polars)
df <- pl$read_parquet('hf://datasets/browndw/coca-ai-parallel-corpus-biber/**/*.parquet')
df <- df$to_data_frame()
Citation
If you use the corpus as part of your research, please cite:
Do LLMs write like humans? Variation in grammatical and rhetorical styles
@misc{reinhart2024llmswritelikehumans,
title={Do… See the full description on the dataset page: https://huggingface.co/datasets/browndw/human-ai-parallel-corpus-docuscope.french-moore-parallel
French → Mooré (Mossi) Parallel Corpus
Machine-translated parallel sentences from French (fr) to Mooré / Mossi (mos), produced by a public-web crawl + filtering + Glosbe translation pipeline.
Snapshot
Field
Value
Validated pairs
3,000,040
Source language
French
Target language
Mooré (Mossi)
Translator
Glosbe public MT
Export date
2026-08-14
Schema
Column
Type
Description
id
string (UUID)
Pair identifier… See the full description on the dataset page: https://huggingface.co/datasets/louisbertson/french-moore-parallel.ParallelKernelBench_Kernels
ParallelKernelBench Kernels
Net-new multi-GPU CUDA kernels generated by LLMs for ParallelKernelBench.
Each subdirectory under solutions/ is one model run. File names match the benchmark problem stems (e.g. 17_rope_allgather_cuda.py ↔ problem 17_rope_allgather in willychan21/ParallelKernelBench_Problems).
Layout
solutions/
<run_id>/
<stem>_cuda.py
...
Runs (1 run(s), 87 kernel files)
run_id
kernels
path… See the full description on the dataset page: https://huggingface.co/datasets/willychan21/ParallelKernelBench_Kernels.french-moore-parallel-conf-ge-0.5
French → Mooré (confidence ≥ 0.5)
Subset of the full French–Mooré validated parallel corpus restricted to pairs with
translation_confidence >= 0.5.
Snapshot
Field
Value
Pairs in this subset
~2.42 million
Filter
translation_confidence >= 0.5
Source language
French
Target language
Mooré (Mossi)
Translator
Glosbe public MT
Parent dataset
full validated export (confidence floor ~0.35)
Files
fr-mos-validated-conf-ge-0.5.jsonl… See the full description on the dataset page: https://huggingface.co/datasets/louisbertson/french-moore-parallel-conf-ge-0.5.quran-parallel-corpus
Quran Parallel Corpus
Verse-aligned Quran parallel corpus — Arabic (Uthmani), English (Sahih International), and Indonesian (Ministry of Religious Affairs).
Stats
Total verses: 6236
Languages: Arabic, English, Indonesian
Translation pairs: Arabic↔English, Arabic↔Indonesian, English↔Indonesian
Formats: JSONL, CSV, Parquet
Structure
Each verse record contains:
Field
Description
surah_number
Chapter (1–114)
surah_name_arabic
Arabic surah… See the full description on the dataset page: https://huggingface.co/datasets/sarjukesumo/quran-parallel-corpus.coca-ai-parallel-corpus-spacy
Citation
If you use the corpus as part of your research, please cite:
Do LLMs write like humans? Variation in grammatical and rhetorical styles
@misc{reinhart2024llmswritelikehumans,
title={Do LLMs write like humans? Variation in grammatical and rhetorical styles},
author={Alex Reinhart and David West Brown and Ben Markey and Michael Laudenbach and Kachatad Pantusen and Ronald Yurko and Gordon Weinberg},
year={2024},
eprint={2410.16107}… See the full description on the dataset page: https://huggingface.co/datasets/browndw/coca-ai-parallel-corpus-spacy.human-ai-parallel-detection
Dataset Card for human-ai-parallel-detection
Dataset Description
Dataset Summary
The human-ai-parallel-detection dataset contains 600 balanced instances for evaluating methods to distinguish between human-written and AI-generated text continuations. Each instance includes a 500-word human-written prompt followed by parallel continuations from humans, GPT-4o, and LLaMA-70B-Instruct. The dataset includes both style embedding features and LLM-as-judge predictions… See the full description on the dataset page: https://huggingface.co/datasets/ephipi/human-ai-parallel-detection.so101_parallel_u01
SO101 Parallel U01 Dataset
Teleoperation dataset collected using SO101 robot with LeRobot.
Robot
Robot: SO101 follower
Teleoperation: SO101 leader
Cameras
Intel RealSense D435i
Intel RealSense D405
Features
This dataset contains:
action
observation.state
observation.images.d435i
observation.images.d405
timestamp
episode_index
frame_index
Task
Pick and place
Dataset Format
LeRobot v3.0 format.… See the full description on the dataset page: https://huggingface.co/datasets/codenmood/so101_parallel_u01.french-moore-parallel
French → Mooré (Mossi) Parallel Corpus
Machine-translated parallel sentences from French (fr) to Mooré / Mossi (mos), produced by a public-web crawl + filtering + Glosbe translation pipeline.
Snapshot
Field
Value
Validated pairs
3,000,040
Source language
French
Target language
Mooré (Mossi)
Translator
Glosbe public MT
Export date
2026-08-14
Schema
Column
Type
Description
id
string (UUID)
Pair identifier… See the full description on the dataset page: https://huggingface.co/datasets/cidjeu/french-moore-parallel.eval_act-lang-policy-test-gp1to3_parallelThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "so100_follower",
"total_episodes": 5,
"total_frames": 1872,
"total_tasks": 1,
"total_videos": 10,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:5"},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/Ayush-Sharan-1/eval_act-lang-policy-test-gp1to3_parallel.sft-mathhard-medium-with-thinking-full-parallelkaa-parallel-corpus
Kaa Karakalpak-English Parallel Corpus (FineTranslations)
📌 Overview
This repository contains a high-quality, curated parallel corpus for the Karakalpak (kaa) language, paired with English (en). Karakalpak is a low-resource Turkic language spoken primarily in the Republic of Karakalpakstan.
This dataset is a specialized subset extracted from the massive HuggingFaceFW/finetranslations project. The goal of this repo is to provide a dedicated and easy-to-access resource… See the full description on the dataset page: https://huggingface.co/datasets/nickoo004/kaa-parallel-corpus.Parallel-Reasoning-Math-thinkbak-rus-parallel
Russian-Bashkir Parallel Corpus
A large-scale parallel corpus for Russian-Bashkir machine translation, containing 9.7M sentence pairs filtered by semantic similarity.
Dataset Details
Languages: Russian (rus) ↔ Bashkir (bak)
Size: 9,768,889 translation pairs
Repository: BorisTM/bak-rus-parallel
Usage
from datasets import load_dataset
dataset = load_dataset("BorisTM/bak-rus-parallel")
Dataset Fields
Field
Type
Description
rus
string… See the full description on the dataset page: https://huggingface.co/datasets/BorisTM/bak-rus-parallel.low_resource_parallel_corpora
The Little Prince — multiparallel corpus (22 languages, RU pivot)
A sentence-level multiparallel corpus of Antoine de Saint-Exupéry's The Little Prince,
built around the classic Russian translation by Nora Gal as the pivot and covering
21 further editions, most of them in low-resource minority languages of Russia.
Every one of the 1,565 pivot sentences has exactly one aligned sentence in every
included language — a perfect N-way alignment (no gaps, no merges). The editions were… See the full description on the dataset page: https://huggingface.co/datasets/averoo/low_resource_parallel_corpora.Parallel-Reasoning-Mathtunisian-msa-parallel-corpus-evaluated
Dataset Description
This dataset is a synthetic parallel corpus of Tunisian Arabic (aeb) and Modern Standard Arabic (arb).
It was created with a rigorous multi-stage pipeline to maximize quality and reproducibility, addressing the scarcity of high-quality resources for Tunisian Arabic NLP.
The primary goals are to support:
Machine translation between Tunisian Arabic and MSA.
Research in dialectal-aware text generation and evaluation.
Cross-dialect representation learning in… See the full description on the dataset page: https://huggingface.co/datasets/tunis-ai/tunisian-msa-parallel-corpus-evaluated.human-ai-parallel-corpus-2-emotionscell3_parallel_gripper_camera_test_20260821This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so-101",
"total_episodes": 1,
"total_frames": 323,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 20,
"splits": {
"train": "0:1"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/pravsels/cell3_parallel_gripper_camera_test_20260821.ccnews-sentences-parallel-splitbso101_airbag_bluebox_parallelThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 15,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/andrejarden/so101_airbag_bluebox_parallel.