datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wmt16
Dataset Card for "wmt16"
Dataset Summary
Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale pertains… See the full description on the dataset page: https://huggingface.co/datasets/wmt/wmt16.WMT-16-PubMedWMT'16 Biomedical Translation Task - PubMed parallel datasets
http://www.statmt.org/wmt16/biomedical-translation-task.htmlwmt16_Qwen3-30B-A3B_moe_patternsMuST-C-and-WMT16-de-enwmt16wmt16-en-ro-pre-processed
WMT16 English-Romanian Translation Data w/ further preprocessing
The original instructions are here.
This pre-processed dataset was created by running:
git clone https://github.com/rsennrich/wmt16-scripts
cd wmt16-scripts
cd sample
./download_files.sh
./preprocess.sh
It was originally used by transformers finetune_trainer.py
The data itself resides at https://cdn-datasets.huggingface.co/translation/wmt_en_ro.tar.gz
If you would like to convert it to jsonlines I've included a small… See the full description on the dataset page: https://huggingface.co/datasets/stas/wmt16-en-ro-pre-processed.wmt16_qwen1.5MoEA2.7B_token_patternsWmt16-TR
Dataset Card for Wmt16-TR
Dataset Description
Wmt16-TR is originally released by TR-MTEB group.
Dataset Structure
We kept the original data structure.
Data Fields
anchor (string): A context passage in Turkish.
positive (string): The anchor's "parallel" sentences in English.
Source Dataset
hf.co/trmteb/wmt16_en_tr_fine_tuning_dataset
wmt16_en_trwmt16_qwen1.5MoEA2.7B_token_real_and_predicted_patterns_t5-smallwmt16_de_en
WMT16_DE_EN
This repository hosts a copy of the WMT16 DE-EN dataset, a parallel corpus used for training and evaluating machine translation systems, specifically focused on the German-English language pair.
WMT16 (Workshop on Machine Translation) DE-EN is part of a larger set of datasets used in the annual WMT machine translation competition. It contains a large collection of sentence pairs in German and their corresponding English translations, often used for evaluating models on… See the full description on the dataset page: https://huggingface.co/datasets/S3IC/wmt16_de_en.wmt16_deepseek-moe-16b-chat_moe_patternswmt16_sentence_lang_en
Dataset Card for "wmt16_sentence_lang_en"
More Information needed
autoeval-eval-wmt16-cs-en-ba4e67-42154145091
Dataset Card for AutoTrain Evaluator
This repository contains model predictions generated by AutoTrain for the following task and dataset:
Task: Translation
Model: Lvxue/finetuned-mt5-small-10epoch
Dataset: wmt16
Config: cs-en
Split: train
To run new evaluation jobs, visit Hugging Face's automatic model evaluator.
Contributions
Thanks to @DarkSourceOfCode for evaluating this model.
wmt16_en_tr_fine_tuning_datasetwmt16-en-ro-extracted
WMT16 English-Romanian Extracted
This dataset is an extracted English-to-Romanian version of the WMT16 Romanian-English dataset available through Hugging Face Datasets.
The original dataset is loaded from:
load_dataset("wmt/wmt16", "ro-en")
wmt16-sampleautoeval-eval-wmt16-cs-en-110a70-42155145092
Dataset Card for AutoTrain Evaluator
This repository contains model predictions generated by AutoTrain for the following task and dataset:
Task: Translation
Model: Lvxue/finetuned-mt5-small-10epoch
Dataset: wmt16
Config: cs-en
Split: train
To run new evaluation jobs, visit Hugging Face's automatic model evaluator.
Contributions
Thanks to @DarkSourceOfCode for evaluating this model.
wmt16_Qwen1.5-MoE-A2.7B-Chat_token_patternswmt16_en_tr_fine_tuning_datasetwmt16_sentence_lang
Dataset Card for "wmt16_sentence_lang"
More Information needed
wmt16_pseudo-labelingautoeval-eval-wmt16-de-en-bfa340-42157145094
Dataset Card for AutoTrain Evaluator
This repository contains model predictions generated by AutoTrain for the following task and dataset:
Task: Translation
Model: Lvxue/finetuned-mt5-small-10epoch
Dataset: wmt16
Config: de-en
Split: test
To run new evaluation jobs, visit Hugging Face's automatic model evaluator.
Contributions
Thanks to @DarkSourceOfCode for evaluating this model.
wmt16_mixtral8x7bInstructv0.1_token_patternswmt16_switch128_token_real_and_predicted_patternsdecode_pattern.shape: (num_layers, num_decoding_steps)
predictor_pattern.shape: (num_decoding_steps, num_layers, top3_indices)
We need to permute predictor_pattern via:
predictor_pattern = predictor_pattern.permute(1,0,2) # (num_layers, num_decoding_steps, top3_indices)
wmt16_deepseek-moe-16b-chat_token_patternswmt16_mixtral8x7bInstructv0.1_token_real_and_predicted_patterns_t5-small_dff2048_dmodel32wmt16_qwen1.5MoEA2.7B_token_patternswmt16_switch64_token_patternswmt16-data