wmt
Datasets
All datasets matching “wmt”wmt_t2t
Dataset Card for "wmt_t2t"
Dataset Summary
The WMT EnDe Translate dataset used by the Tensor2Tensor library.
Translation dataset based on the data from statmt.org.
Versions exist for different years using a combination of data
sources. The base wmt allows you to create a custom dataset by choosing
your own data/language pair. This can be done as follows:
from datasets import inspect_dataset, load_dataset_builder
inspect_dataset("wmt_t2t", "path/to/scripts")
builder =… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt_t2t.wmt24pp
WMT24++
This repository contains the human translation and post-edit data for the 55 en->xx language pairs released in
the publication
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects.
If you are interested in the MT/LLM system outputs and automatic metric scores, please see MTME.
If you are interested in the images of the source URLs for each document, please see here.
Schema
Each language pair is stored in its own jsonl file.
Each row… See the full description on the dataset page: https://huggingface.co/datasets/google/wmt24pp.wmt19
Dataset Card for "wmt19"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt19.wmt14
Dataset Card for "wmt14"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt14.wmt16
Dataset Card for "wmt16"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt16.wmt18
Dataset Card for "wmt18"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt18.