wmt16
Datasets
All datasets matching “wmt16”wmt16
Dataset Card for "wmt16"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt16.WMT-16-PubMedWMT'16 Biomedical Translation Task - PubMed parallel datasets
http://www.statmt.org/wmt16/biomedical-translation-task.htmlwmt16_Qwen3-30B-A3B_moe_patternsMuST-C-and-WMT16-de-enwmt16wmt16-en-ro-pre-processed
WMT16 English-Romanian Translation Data w/ further preprocessing
The original instructions are here.
This pre-processed dataset was created by running:
git clone https://github.com/rsennrich/wmt16-scripts
cd wmt16-scripts
cd sample
./download_files.sh
./preprocess.sh
It was originally used by transformers finetune_trainer.py
The data itself resides at https://cdn-datasets.huggingface.co/translation/wmt_en_ro.tar.gz
If you would like to convert it to jsonlines I've included a small… See the full description on the dataset page: https://huggingface.co/datasets/stas/wmt16-en-ro-pre-processed.