datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wmt_t2t
Dataset Card for "wmt_t2t"
Dataset Summary
The WMT EnDe Translate dataset used by the Tensor2Tensor library.
Translation dataset based on the data from statmt.org.
Versions exist for different years using a combination of data
sources. The base wmt allows you to create a custom dataset by choosing
your own data/language pair. This can be done as follows:
from datasets import inspect_dataset, load_dataset_builder
inspect_dataset("wmt_t2t", "path/to/scripts")
builder =… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt_t2t.wmt24pp
WMT24++
This repository contains the human translation and post-edit data for the 55 en->xx language pairs released in
the publication
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects.
If you are interested in the MT/LLM system outputs and automatic metric scores, please see MTME.
If you are interested in the images of the source URLs for each document, please see here.
Schema
Each language pair is stored in its own jsonl file.
Each row… See the full description on the dataset page: https://huggingface.co/datasets/google/wmt24pp.wmt19
Dataset Card for "wmt19"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt19.wmt14
Dataset Card for "wmt14"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt14.wmt16
Dataset Card for "wmt16"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt16.wmt18
Dataset Card for "wmt18"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt18.wmt17
Dataset Card for "wmt17"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt17.wmt15
Dataset Card for "wmt15"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt15.wmt20_mlqe_task1
Dataset Card for WMT20 - MultiLingual Quality Estimation (MLQE) Task1
Dataset Summary
From the homepage:
This shared task (part of WMT20) will build on its previous editions to further examine automatic methods for estimating the quality of neural machine translation output at run-time, without relying on reference translations. As in previous years, we cover estimation at various levels. Important elements introduced this year include: a new task where sentences are… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt20_mlqe_task1.wmt-mqm-error-spans
Dataset Summary
This dataset contains all MQM human annotations from previous WMT Metrics shared tasks and the MQM annotations from Experts, Errors, and Context in a form of error spans. Moreover, it contains some hallucinations used in the training of XCOMET models.
Please note that this is not an official release of the data and the original data can be found here.
The data is organised into 8 columns:
src: input text
mt: translation
ref: reference translation
annotations: List… See the full description on the dataset page: https://huggingface.co/datasets/RicardoRei/wmt-mqm-error-spans.wmt19
Dataset Card for wmt19
This is a preprocessed version of wmt19 dataset for benchmarks in LM-Polygraph.
Dataset Details
Dataset Description
Curated by: https://huggingface.co/LM-Polygraph
License: https://github.com/IINemo/lm-polygraph/blob/main/LICENSE.md
Dataset Sources [optional]
Repository: https://github.com/IINemo/lm-polygraph
Uses
Direct Use
This dataset should be used for performing benchmarks on… See the full description on the dataset page: https://huggingface.co/datasets/LM-Polygraph/wmt19.WMT-month-splitswmt19
Dataset Card for "wmt19"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale… See the full description on the dataset page: https://huggingface.co/datasets/WatsonNT/wmt19.wmt-human-all-TTS
WMT Human + TTS Audio
WMT human evaluation data (zouharvi/wmt-human-all) extended with TTS-synthesised source audio, covering 49 language pairs. Used as training data for SpeechCOMET.
Part of the SpeechCOMET model family | Paper: Why We Need Speech to Evaluate Speech Translation (Züfle et al., 2026) | Code: github.com/MaikeZuefle/speechCOMET
Dataset
Each row contains a source sentence, a machine translation hypothesis, a human quality score, and TTS-synthesised source… See the full description on the dataset page: https://huggingface.co/datasets/maikezu/wmt-human-all-TTS.wmt24pp-qtranslated
WMT24++ Translated by Quantized LLMs (wmt24pp-qtranslated)
This dataset accompanies the paper: [TBC]
It provides segment‑level translations and metadata for 55 languages × 110 directions produced by variants of the Llama 3.x and Qwen3 model families, each quantized with up to four post‑training quantization (PTQ) methods and two bit‑widths.
🌍 Dataset Structure
Split names : <src>-<tgt> (e.g. ar_EG-en, en-zu_ZA) ─ 110 in total
Columns :
• source_segment… See the full description on the dataset page: https://huggingface.co/datasets/bnjmnmarie/wmt24pp-qtranslated.wmt20_mlqe_task2
Dataset Card for WMT20 - MultiLingual Quality Estimation (MLQE) Task2
Dataset Summary
From the homepage:
This shared task (part of WMT20) will build on its previous editions to further examine automatic methods for estimating the quality of neural machine translation output at run-time, without relying on reference translations. As in previous years, we cover estimation at various levels. Important elements introduced this year include: a new task where sentences are… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt20_mlqe_task2.wmt-da-human-evaluation-long-context
Dataset Summary
Long-context / document-level dataset for Quality Estimation of Machine Translation.
It is an augmented variant of the sentence-level WMT DA Human Evaluation dataset.
In addition to individual sentences, it contains augmentations of 2, 4, 8, 16, and 32 sentences, among each language pair lp and domain.
The raw column represents a weighted average of scores of augmented sentences using character lengths of src and mt as weights.
The code used to apply the augmentation… See the full description on the dataset page: https://huggingface.co/datasets/ymoslem/wmt-da-human-evaluation-long-context.wmt24pp
WMT24++
This repository contains the human translation and post-edit data for the 55 en->xx language pairs released in
the publication
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects.
If you are interested in the MT/LLM system outputs and automatic metric scores, please see MTME.
If you are interested in the images of the source URLs for each document, please see here.
Schema
Each language pair is stored in its own jsonl file.
Each row is… See the full description on the dataset page: https://huggingface.co/datasets/Lulu19971017/wmt24pp.wmt19wmt24pp
WMT24++
This repository contains the human translation and post-edit data for the 55 en->xx language pairs released in
the publication
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects.
If you are interested in the MT/LLM system outputs and automatic metric scores, please see MTME.
If you are interested in the images of the source URLs for each document, please see here.
Schema
Each language pair is stored in its own jsonl file.
Each row is… See the full description on the dataset page: https://huggingface.co/datasets/synquid/wmt24pp.WMT-year-splits
Dataset Card for "WMT-year-splits"
More Information needed
WMT23-Test
Dataset Card for "WMT23-Test"
More Information needed
wmt-mqm-human-evaluation
Dataset Summary
This dataset contains all MQM human annotations from previous WMT Metrics shared tasks and the MQM annotations from Experts, Errors, and Context.
The data is organised into 8 columns:
lp: language pair
src: input text
mt: translation
ref: reference translation
score: MQM score
system: MT Engine that produced the translation
annotators: number of annotators
domain: domain of the input text (e.g. news)
year: collection year
You can also find the original data here.… See the full description on the dataset page: https://huggingface.co/datasets/RicardoRei/wmt-mqm-human-evaluation.wmt-en-fr
Dataset Card for "wmt-en-fr"
More Information needed
wikititleswmt22_african
Dataset Card for allenai/wmt22_african
Dataset Summary
This dataset was created based on metadata for mined bitext released by Meta AI. It contains bitext for 248 pairs for the African languages that are part of the 2022 WMT Shared Task on Large Scale Machine Translation Evaluation for African Languages.
How to use the data
There are two ways to access the data:
Via the Hugging Face Python datasets library
from datasets import load_dataset
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/allenai/wmt22_african.WMT-16-PubMedWMT'16 Biomedical Translation Task - PubMed parallel datasets
http://www.statmt.org/wmt16/biomedical-translation-task.htmlWMT22-Test
Dataset Card for "WMT22-Test"
More Information needed
wmt-da-human-evaluation
Dataset Summary
This dataset contains all DA human annotations from previous WMT News Translation shared tasks.
The data is organised into 8 columns:
lp: language pair
src: input text
mt: translation
ref: reference translation
score: z score
raw: direct assessment
annotators: number of annotators
domain: domain of the input text (e.g. news)
year: collection year
You can also find the original data for each year in the results section https://www.statmt.org/wmt{YEAR}/results.html… See the full description on the dataset page: https://huggingface.co/datasets/RicardoRei/wmt-da-human-evaluation.wmt20_mlqe_task3
Dataset Card for WMT20 - MultiLingual Quality Estimation (MLQE) Task3
Dataset Summary
From the homepage:
This shared task (part of WMT20) will build on its previous editions to further examine automatic methods for estimating the quality of neural machine translation output at run-time, without relying on reference translations. As in previous years, we cover estimation at various levels. Important elements introduced this year include: a new task where sentences are… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt20_mlqe_task3.