datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wmt24pp
WMT24++
This repository contains the human translation and post-edit data for the 55 en->xx language pairs released in
the publication
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects.
If you are interested in the MT/LLM system outputs and automatic metric scores, please see MTME.
If you are interested in the images of the source URLs for each document, please see here.
Schema
Each language pair is stored in its own jsonl file.
Each row… See the full description on the dataset page: https://huggingface.co/datasets/google/wmt24pp.wmt-mqm-error-spans
Dataset Summary
This dataset contains all MQM human annotations from previous WMT Metrics shared tasks and the MQM annotations from Experts, Errors, and Context in a form of error spans. Moreover, it contains some hallucinations used in the training of XCOMET models.
Please note that this is not an official release of the data and the original data can be found here.
The data is organised into 8 columns:
src: input text
mt: translation
ref: reference translation
annotations: List… See the full description on the dataset page: https://huggingface.co/datasets/RicardoRei/wmt-mqm-error-spans.wmt24pp
WMT24++
This repository contains the human translation and post-edit data for the 55 en->xx language pairs released in
the publication
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects.
If you are interested in the MT/LLM system outputs and automatic metric scores, please see MTME.
If you are interested in the images of the source URLs for each document, please see here.
Schema
Each language pair is stored in its own jsonl file.
Each row is… See the full description on the dataset page: https://huggingface.co/datasets/Lulu19971017/wmt24pp.wmt24pp
WMT24++
This repository contains the human translation and post-edit data for the 55 en->xx language pairs released in
the publication
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects.
If you are interested in the MT/LLM system outputs and automatic metric scores, please see MTME.
If you are interested in the images of the source URLs for each document, please see here.
Schema
Each language pair is stored in its own jsonl file.
Each row is… See the full description on the dataset page: https://huggingface.co/datasets/synquid/wmt24pp.AfriMTE-WMT2024
AfriMTE-WMT2024: African Machine Translation Evaluation Challenge Set
Dataset Description
The AfriMTE-WMT2024 dataset is a challenge set used in the WMT 2024 Metrics Shared Task for evaluating machine translation quality across 13 African-centric language pairs. This dataset aims to support research and development in African machine translation evaluation and quality estimation.
Supported Language Pairs
Config
Source Language
Target Language
#… See the full description on the dataset page: https://huggingface.co/datasets/masakhane/AfriMTE-WMT2024.wmt25wmt-human-allThis dataset is continuously updated and contains a compilation of human translation quality assessment from past WMT campaigns.
Specifically, this dataset merges all annotation protocols (DA, MQM, ESA) on a semi-unified scale (0 to 100).
The current version of the dataset includes human scores up to WMT 2025 (inclusive) and has been created with the following script:
import subset2evaluate # version 1.0.20
import json
import statistics
data =… See the full description on the dataset page: https://huggingface.co/datasets/zouhar/wmt-human-all.wmt26-mist-sample
Update Log
22 June 2026 (latest) - we updated our data mix because some BELEBELE samples did not have the context. If you downloaded data before 22 June, please download the new version.
16 June 2026 - first version
Summary
The wmt26-mist-sample is a multilingual mix provided by the WMT26 MIST shared task organizers as a starting point for fine-tuning multilingual LLMs. It contains three types of tasks, to cover same-language and cross-lingual comprehension and… See the full description on the dataset page: https://huggingface.co/datasets/pinzhenchen/wmt26-mist-sample.wmt-terminology-2023
WMT 2023 Terminology Shared Task Data
The current version contains both the sources, references, terminologies but also participant submissions.
It is easily accessible from huggingface and contains only the test split:
from datasets import load_dataset
data = load_dataset("zouharvi/wmt-terminology-2023")["test"]
print(len(data))
If you use this data, please read the paper and cite:
@inproceedings{semenov-etal-2023-findings,
title = "Findings of the WMT 2023 Shared Task on… See the full description on the dataset page: https://huggingface.co/datasets/zouhar/wmt-terminology-2023.wmt26-lt3-sorbian-data
LT3 at WMT26 (Sorbian): reproduction datasets
Companion data repo for the LT3 (Ghent University) WMT26 Sorbian system-description paper.
It carries the datasets needed to reproduce the submitted systems that cannot be fetched
automatically from an online source. Auto-fetchable inputs are deliberately excluded:
dictionaries, the organizer train/dev/test distribution, and raw public corpora already pinned
in the companion repo's data/MANIFESTS/.
Model weights:… See the full description on the dataset page: https://huggingface.co/datasets/TomMoeras/wmt26-lt3-sorbian-data.wmt24pp-kr-reversed
WMT24++ Parallel Mix (Direction-Flipped)
Source dataset dir: /Users/inertia/Desktop/preprocessing/processed/wmt24pp_parallel_80_20
Transformation: direction flip per row (source/target swap)
Pair+sample_key overlap with source: 0
Primary(stats): {'en->ko': 450, 'ko->en': 449}
Auxiliary(flipped): ja->en, en->ja, zh->en, en->zh
Target primary ratio: 0.8000
Achieved primary ratio: 0.7998
Tag template: <{tgt_upper}>
Text template: {src_tag} {source} {tgt_tag} {target}… See the full description on the dataset page: https://huggingface.co/datasets/alwaysgood/wmt24pp-kr-reversed.wmt20-markables
WMT20 Markables
This dataset contains human annotated translation quality of terminologies/markables.
Read the paper here and please cite it as:
@inproceedings{zouhar2020wmt20,
title={WMT20 Document-Level Markable Error Exploration},
author={Zouhar, Vil{\'e}m and Vojt{\v{e}}chov{\'a}, Tereza and Bojar, Ond{\v{r}}ej},
booktitle={Proceedings of the Fifth Conference on Machine Translation},
pages={371--380},
year={2020},
url={https://aclanthology.org/2020.wmt-1.41}
}… See the full description on the dataset page: https://huggingface.co/datasets/zouhar/wmt20-markables.wmt16_de_en
WMT16_DE_EN
This repository hosts a copy of the WMT16 DE-EN dataset, a parallel corpus used for training and evaluating machine translation systems, specifically focused on the German-English language pair.
WMT16 (Workshop on Machine Translation) DE-EN is part of a larger set of datasets used in the annual WMT machine translation competition. It contains a large collection of sentence pairs in German and their corresponding English translations, often used for evaluating models on… See the full description on the dataset page: https://huggingface.co/datasets/S3IC/wmt16_de_en.wmt24pp_xx_to_pt_10k
WMT24PP XX to PT 10k
This is a dataset of translations from various source languages into Portuguese. The pairs are derived from google/wmt24pp, where the same 998 English sentences are each translated into many target languages. For every sentence, we take its translation in another language as the source and its Portuguese translation as the target, yielding {source language}→Portuguese pairs across many languages. Note that both sides are translations of the… See the full description on the dataset page: https://huggingface.co/datasets/amalia-llm/wmt24pp_xx_to_pt_10k.wmt24pp-kr
WMT24++ Parallel Mix
Primary(all): en->ko, ko->en
Primary mode: disjoint_halves
Auxiliary(sampled, no oversampling): en->ja, ja->en, en->zh, zh->en
Target primary ratio: 0.8000
Achieved primary ratio: 0.7998
Tag template: <{tgt_upper}>
Text template: {src_tag} {source} {tgt_tag} {target}
Normalize doubled quotes: True
Strip control chars: True
Drop bad source: True
Drop canary: True
Drop @user handles: True
Drop one-word sentences: True
Columns
id, dataset, pair_config… See the full description on the dataset page: https://huggingface.co/datasets/alwaysgood/wmt24pp-kr.wmt26-mist-test
WMT26 MIST Shared Task Test Release
Submit by 01 August 2026, AoE. For more details, please refer to our website.
Update log
20 July 2026: We fixed an issue where some English QA-OEG prompts are not localised. If you downloaded the data on or before 20 July, please kindly re-download it just in case.
15 July 2026: We fixed an issue where the English QA-OEG prompts are empty.
14 July 2026: Test set release: 24 languages and 3 sub-tasks.
Test data… See the full description on the dataset page: https://huggingface.co/datasets/pinzhenchen/wmt26-mist-test.wmt24pp-ce
WMT24++ Reference Translations for Chechen
Description
WMT24++ benchmark in Chechen
Original WMT24++ benchmark (55 languages): https://huggingface.co/datasets/google/wmt24pp
The reference translation have been created by human translator based on the Russian version of WMT24++
The dataset uses both cases of Cyrillic Palochka Letter where it is grammatically correct.
For preparation of the Chechen version of the dataset we hired a professional native speaker… See the full description on the dataset page: https://huggingface.co/datasets/NM-development/wmt24pp-ce.eseu-wmt26-automotive-trainGerAlpacaDataCleaned_transformer.wmt19.en-deeseu-wmt26-automotive-devSEGALE-WMT24
SEGALE‑WMT24: Long‑Context MT Evaluation Dataset
SEGALE‑WMT24 packages system outputs for three language directions together with controlled synthetic setups. It is designed for the SEGALE scheme, which extends sentence‑level metrics to long documents via segmentation + alignment before scoring, and tracks under/over‑translation via a null‑alignment ratio (NA ratio).
Intended use
Long-context MT benchmarking and meta‑evaluation.
Studying sensitivity of metrics and… See the full description on the dataset page: https://huggingface.co/datasets/rl-bandits-lab/SEGALE-WMT24.wmt24pp-en-bn
WMT24++ English-Bengali Filtered Subset
This dataset is a filtered subset of google/wmt24pp using en-bn_IN.jsonl as the source file.
Filtering rules:
keep rows where is_bad_source is false
keep rows with non-empty source
keep rows with non-empty target
Result:
input rows: 998
kept rows: 960
removed rows: 38
Columns are preserved from the source JSONL.
eseu-wmt26-energy-devQE-wmt21-1k-prometheus
data: WMT21 Metrics Shared Task EN-DE validation set (1k)
QE model: prometheus-eval/prometheus-7b-v2.0
eseu-wmt26-energy-syntheticen-si-translation-wmt-internet-1300
En Si Translation Wmt Internet 1300
Dataset Summary
English-Sinhala Web Forum and Internet Translation dataset containing ~1,300 high-quality sentences capturing internet slang and long-form narrative text from WMT20.
Engineering Pipeline Parameters
Language Pair: English (en) to Sinhala (si)
Total Valid Token Rows: 1300
Internal Storage Structure: Single-File data.json
Upstream Source Attribution
This specific sub-split was compiled and extracted… See the full description on the dataset page: https://huggingface.co/datasets/SAWithanage/en-si-translation-wmt-internet-1300.eseu-wmt26-automotive-syntheticwmt14_en-de
SmolTrans EN-DE Dataset
High-quality English–German parallel dataset filtered using:
Language detection
HTML removal
Duplicate removal
LM perplexity filtering
Perplexity ratio filtering
Embedding cosine similarity
Statistics
Total pairs: 4.5M
Median length: ...
Filtering removed: ~X%
Format
{"en": "...", "de": "..."}
Usage
from datasets import load_dataset
ds = load_dataset("/smoltrans-en-de")
Notes
Short sentences may show unstable… See the full description on the dataset page: https://huggingface.co/datasets/Neural-Navigator-Labs/wmt14_en-de.wmt18-cs-en-preprocessed---
language:
- cs
- en
task_categories:
- translation
pretty_name: WMT18 Czech-English Preprocessed
size_categories:
- 10K<n<100K
---
# WMT18 Czech-English Preprocessed
This dataset is a preprocessed subset of the WMT18 Czech-English translation dataset.
Original dataset: https://huggingface.co/datasets/wmt/wmt18
## Dataset Description
The dataset contains Czech-English parallel sentence pairs for machine translation.
Each example contains one Czech sentence and its corresponding… See the full description on the dataset page: https://huggingface.co/datasets/charlie0831/wmt18-cs-en-preprocessed.eseu-wmt26-energy-train