datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
translateplus-flores-benchmark
TranslatePlus Translation Benchmark (FLORES 2026)
This dataset contains benchmark results for TranslatePlus Translation API across 20 global languages using the FLORES dataset.
👉 Try the API: https://translateplus.io
Methodology
Dataset: FLORES (Facebook)
Samples per language: 997
Source language: English
Target languages: Top 20 global languages
Evaluation metrics:
BLEU (sacreBLEU)
COMET (Unbabel/wmt22-comet-da)
Evaluation type: Reference-based (human… See the full description on the dataset page: https://huggingface.co/datasets/meetsohail/translateplus-flores-benchmark.flores-parallelflores_plus_gender
FLORES+Gender
This dataset builds on the FLORES+ benchmark, developed by Meta to assess machine translation (MT) systems for low-resource languages. FLORES+Gender is designed to assess gender bias in MT. While the typical approach examines bias by translating from a genderless language into a gendered one, this dataset follows the methodology of Costa-jussà et al. (2023) and reverses the direction to analyse whether translation quality is affected by the predominant grammatical… See the full description on the dataset page: https://huggingface.co/datasets/HiTZ/flores_plus_gender.aihub-flores-koen-integrated-prime-small-30k
High Quality Ko-En Translation Dataset (AIHub-FLoRes Integrated)
AI Hub의 한-영 번역 데이터셋과 FLoRes 한-영 번역 데이터셋의 합본입니다.
High Quality AIHub Dataset
AI Hub의 경우 한-영 번역 관련 데이터셋을 8개 병합한 병렬 데이터 traintogpb/aihub-koen-translation-integrated-tiny-100k에서 고품질의 번역 레퍼런스를 가진 데이터만 추출하였습니다.
번역 레퍼런스 품질 평가 척도는 Unbabel/XCOMET-XL (3.5B)로 측정한 xCOMET metric입니다.
8개의 AIHub 데이터 소스 중 기존 실험을 통해 번역 성능(SacreBLEU)이 낮았던 4개의 소스에서 xCOMET 기준 상위 5,000개, 그 외 4개의 소스에서 xCOMET 기준 상위 2,500개를 추출해 총 약 3만 개의 데이터를… See the full description on the dataset page: https://huggingface.co/datasets/traintogpb/aihub-flores-koen-integrated-prime-small-30k.2M-Flores-ASL
2M-Flores
As part of the 2M-Belebele project, we have produced video recodings of ASL signing for all the dev and devtest
sentences in the original flores200 dataset.
To obtain ASL sign recordings, we provide translators of ASL and native signers with the English text version of the sentences to be recorded.
The interpreters are then asked to translate these sentences into ASL, create glosses for all sentences, and record their interpretations into ASL one sentence at a time.
The… See the full description on the dataset page: https://huggingface.co/datasets/alj68/2M-Flores-ASL.flores_plusaihub-flores-koen-integrated-prime-base-300k
High Quality Ko-En Translation Dataset (AIHub-FLoRes Integrated)
AI Hub의 한-영 번역 데이터셋과 FLoRes 한-영 번역 데이터셋의 합본입니다.
High Quality AIHub Dataset
AI Hub의 경우 한-영 번역 관련 데이터셋을 8개 병합한 병렬 데이터 traintogpb/aihub-koen-translation-integrated-mini-1m에서 고품질의 번역 레퍼런스를 가진 데이터만 추출하였습니다.
번역 레퍼런스 품질 평가 척도는 Unbabel/XCOMET-XL (3.5B)로 측정한 xCOMET metric입니다.
8개의 AIHub 데이터 소스의 구성 비율은 실험을 통해 확보한 번역 성능(SacreBLEU)에 따라 차등을 두었습니다.
FLoRes Dataset
FLoRes-200 데이터셋의 경우 997개의 dev, 1,012개의… See the full description on the dataset page: https://huggingface.co/datasets/traintogpb/aihub-flores-koen-integrated-prime-base-300k.FLORES200_translations_GPT4
Dataset Summary
This dataset consists of three synthetic parallel English-to-Faroese translations of 1,012 sentences from the FLORES-200 benchmark. The translations were generated using GPT-4 Turbo (gpt-4-1106-preview) with three different prompting strategies:
Zero-shot translation (no additional examples provided).
Random few-shot translation (12 few-shot examples selected randomly).
STS-based few-shot translation (12 few-shot examples selected using Semantic Textual Similarity).… See the full description on the dataset page: https://huggingface.co/datasets/barbaroo/FLORES200_translations_GPT4.Flores-subsetflores-madar-inference