datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Multilingual-Thinking
Dataset summary
Multilingual-Thinking is a reasoning dataset where the chain-of-thought has been translated from English into one of 4 languages: Spanish, French, Italian, and German. The dataset was created by sampling 1k training samples from the SystemChat subset of SmolTalk2 and translating the reasoning traces with another language model.
This dataset was used in the OpenAI Cookbook to fine-tune the OpenAI gpt-oss models.
You can load the dataset using:
from datasets import… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking.multilingual_mmluMMLU professionally translated into 14 languages using professional human translators, sourced from OpenAI's simple-eval.
Original files:
english: https://openaipublic.blob.core.windows.net/simple-evals/mmlu.csv
multilingual: https://openaipublic.blob.core.windows.net/simple-evals/mmlu_{language}.csv where language one of "AR-XY", "BN-BD", "DE-DE", "ES-LA", "FR-FR", "HI-IN", "ID-ID", "IT-IT", "JA-JP", "KO-KR", "PT-BR", "ZH-CN", "SW-KE", "YO-NG", "EN-US"
McEvalMcEval benchmark data as described in the McEval Paper. Code for the evaluation can be found on Github as McEval.
Nemotron-SFT-Multilingual-v2
Dataset Description:
Nemotron-SFT-Multilingual-v2 is a multilingual supervised fine-tuning (SFT) dataset for post-training text-generation models. It is generated by translating seed data from Nemotron-Math-v2, Nemotron-Competitive-Programming-v1, and Nemotron-Science-v1, adding multilingual coverage for Hindi (hi), Korean (ko), Brazilian Portuguese (pt-br), and refreshed Japanese (ja) data.
The dataset is generated with a new data processing pipeline that avoids line-breaking… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-SFT-Multilingual-v2.hle-multilingual
HLE Multilingual
Multilingual translations of HLE (Humanity's Last Exam), an expert-level QA benchmark with questions across math, science, humanities, and engineering designed to challenge even domain experts.
Source: cais/hle (test split, 2,158 text-only questions out of 2,500 total)
Languages
Config
Language
Examples
ces
Czech
50
dan
Danish
50
deu
German
800
fin
Finnish
50
fra
French
50
ita
Italian
50
nld
Dutch
50
pol
Polish
50
spa… See the full description on the dataset page: https://huggingface.co/datasets/ellamind/hle-multilingual.gsm8k-platinum-multilingual
GSM8K Platinum Multilingual
Multilingual translations of GSM8K Platinum, a rigorously cleaned and verified version of GSM8K containing 1,209 elementary math word problems requiring multi-step arithmetic reasoning.
Source: madrylab/gsm8k-platinum (test split, 1,209 questions)
Languages
Config
Language
Examples
ces
Czech
100
dan
Danish
100
deu
German
1,209
fin
Finnish
100
fra
French
100
ita
Italian
100
nld
Dutch
100
pol
Polish
100
spa
Spanish… See the full description on the dataset page: https://huggingface.co/datasets/ellamind/gsm8k-platinum-multilingual.tiny-aya-l2-thinker-multilingual-reasoning
Tiny Aya L2 Multilingual Reasoning (44 languages)
Translated multilingual reasoning traces used to train Tiny Aya L2-Thinker.
Each example has the prompt, thinking, and answer in the same non-English language alongside the original texts in English.
Data source
Prompts from AM-DeepSeek-R1-0528-Distilled
Thinking traces and outputs distilled from gpt-oss-120b
Translated with command-a-translate and DeepSeek-V3
Languages (44)
Language
Train… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/tiny-aya-l2-thinker-multilingual-reasoning.Dolci-Think-SFT-32B-Multilingual
Dolci-Think-SFT-32B-Multilingual
Dolci-Think-SFT-32B-Multilingual is a large-scale multilingual long chain-of-thought (CoT) reasoning corpus spanning six languages: English, French, German, Spanish, Chinese, and Swahili. Each sample includes a question, a long-form reasoning trace, and a final answer, all translated into the target language, with sequences up to 32,768 tokens.
It is released alongside the paper Rethinking the Multilingual Reasoning Gap with Layer Swap.… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/Dolci-Think-SFT-32B-Multilingual.openai_multilingual_mmluMMLU professionally translated into 14 languages using professional human translators, sourced from OpenAI's simple-eval.
Original files:
english: https://openaipublic.blob.core.windows.net/simple-evals/mmlu.csv
multilingual: https://openaipublic.blob.core.windows.net/simple-evals/mmlu_{language}.csv where language one of "AR-XY", "BN-BD", "DE-DE", "ES-LA", "FR-FR", "HI-IN", "ID-ID", "IT-IT", "JA-JP", "KO-KR", "PT-BR", "ZH-CN", "SW-KE", "YO-NG", "EN-US"
gsm8k-multilingual-reasoning
gsm8k-multilingual-reasoning
GSM8K with reasoning translated to multiple languages
Schema
{"prompt": "...", "answer": "...", "reasoning": "...", "metadata": {...}}
Usage
from datasets importload_dataset
ds = load_dataset("eddie-OB/gsm8k-multilingual-reasoning")
print(ds["train"][0])
Source
Derived from OpenAI GSM8K.
multilingual-sycophancy
Multilingual Sycophancy
A Parallel Benchmark for Cross-Lingual Alignment Failure across 38 Languages, 33 Opinion Categories, and 3 Resource Tiers.
This dataset accompanies the research paper Sycophancy as a Multilingual Alignment Failure: How Safety Degrades Across Languages, Topics, and Models. It contains 188,100 parallel records (4,950 per language × 38 languages) — each a triple of (prompt, sycophantic response, non-sycophantic response) — designed for forced-choice… See the full description on the dataset page: https://huggingface.co/datasets/aryashah00/multilingual-sycophancy.databricks-dolly-15k-curated-multilingual
Dataset Card for "databricks-dolly-15k-curated-multilingual"
A curated and multilingual version of the Databricks Dolly instructions dataset. It includes a programmatically and manually corrected version of the original en dataset. See below.
STATUS:
Currently, the original Dolly v2 English version has been curated combining automatic processing and collaborative human curation using Argilla (~400 records have been manually edited and fixed). The following graph shows a summary… See the full description on the dataset page: https://huggingface.co/datasets/argilla/databricks-dolly-15k-curated-multilingual.c4_urls_multilingual
Dataset Card for c4_urls_multilingual
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (multilingual variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_multilingual.african-multilingual-tokenizer-challenge
African Multilingual Tokenizer Challenge dataset
The frozen public corpus for the African Multilingual Tokenizer Challenge. It contains one balanced multilingual train split and one balanced validation split.
Split
Per language
Total
Train
40,000
240,000
Validation
4,000
24,000
Languages are English (en), French (fr), Hausa (ha), Swahili (sw), Yoruba (yo) and Amharic (am). Official public-test and private-test text are deliberately absent from this repository.… See the full description on the dataset page: https://huggingface.co/datasets/Similoluwa/african-multilingual-tokenizer-challenge.gsm8k-multilingual
gsm8k-multilingual
GSM8K translated to multiple languages (no reasoning)
Schema
{"prompt": "...", "answer": "...", "metadata": {...}}
Usage
from datasets import load_dataset
ds = load_dataset("eddie-OB/gsm8k-multilingual")
print(ds["train"][0])
Source
Derived from OpenAI GSM8K.
gpqa-multilingual
GPQA Multilingual
Multilingual translations of GPQA (Graduate-Level Google-Proof Q&A), a challenging multiple-choice benchmark requiring graduate-level expertise in biology, physics, and chemistry.
Source: Idavidrein/gpqa (gpqa_main, 448 questions)
Languages
Config
Language
Examples
ces
Czech
448
dan
Danish
448
deu
German
448
fin
Finnish
50
fra
French
448
ita
Italian
448
nld
Dutch
448
pol
Polish
448
spa
Spanish
448
More to be added later.… See the full description on the dataset page: https://huggingface.co/datasets/ellamind/gpqa-multilingual.multilingual-text
Multilingual Text Dataset
This dataset contains a curated selection of rows from multiple input datasets, where each row includes a text chunk of approximately 2000 tokens (as measured by Llama 3.1 tokenizer) verified to be written in the correct language. Only rows with properly classified language chunks are retained, ensuring high-quality multilingual data for analysis or model training.
Preprocessing Steps
Normalized whitespace, punctuation, Unicode characters, and… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/multilingual-text.simpleqa-verified-multilingual
SimpleQA Verified Multilingual
Multilingual translations of SimpleQA Verified, a 1,000-prompt factuality benchmark from Google DeepMind that evaluates short-form parametric knowledge (facts stored in model weights).
Source: google/simpleqa-verified (eval split, 1,000 examples)
Languages
Config
Language
Examples
ces
Czech
100
dan
Danish
100
deu
German
1,000
fra
French
100
ita
Italian
100
nld
Dutch
100
pol
Polish
100
spa
Spanish
100
More to… See the full description on the dataset page: https://huggingface.co/datasets/ellamind/simpleqa-verified-multilingual.multilingual_TinyStories
Dataset Card for Multilingual TinyStories
Dataset Details
Dataset Description
The Multilingual TinyStories dataset contains translations of the original TinyStories dataset, which consists of synthetically generated short stories using a small vocabulary suitable for 3 to 4-year-olds. These stories were originally generated by GPT-3.5 and GPT-4. The multilingual versions have been translated into various languages, including Spanish, Chinese, German, Turkish… See the full description on the dataset page: https://huggingface.co/datasets/Gabrui/multilingual_TinyStories.high-quality-multilingual-sentences
High Quality Multilingual Sentences
This dataset contains multilingual sentences derived from the agentlans/LinguaNova dataset.
It includes 1.58 million rows across 51 different languages, each in its own configuration.
Example row (from the all config):
{
"text": "امام جمعه اصفهان گفت: میزان نیاز آب شرب اصفهان ۱۱.۵ متر مکعب است که تمام استان اصفهان را پوشش میدهد و نسبت به قبل از انقلاب یکی از پیشرفتها در حوزه آب بوده است.",
"fasttext": "fa",
"gcld3": "fa"
}
Fields:… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/high-quality-multilingual-sentences.risale-nur-multilingual
Risale-i Nur Multilingual Corpus
Bediüzzaman Said Nursî'nin Risale-i Nur külliyatının 27 dilde çok dilli korpusu — her eser başlıklara göre bölümlere (section) ayrılmış, bölümler diller arasında hizalanmış ve konu (topic) hiyerarşisiyle etiketlenmiştir.
Güncel release: v2.10.0 · 20 config/lane · 163,820 config-split satırı. Alt başlıklardaki eski v2.x etiketleri lane'in ilk eklendiği sürümü gösterir; güncel release sürümü değildir. Deterministik projeksiyonlar duplicate_of ile… See the full description on the dataset page: https://huggingface.co/datasets/risaleinur/risale-nur-multilingual.McEval-InstructMcEval-Instruct data as described in the McEval Paper. Code for the evaluation and sft can be found on Github as McEval.
TinyStories-Multilingual
Novelist: TinyStories Multilingual Edition
Dataset Summary
The TinyStories Multilingual Edition is a high-fidelity synthetic dataset of short, child-safe fiction designed to stress-test literary consistency, emotional warmth, and multilingual fluency in small models. Derived from the broader Novelist ecosystem, this subset focuses on narrative simplicity paired with complex moral and social themes.
The dataset contains 15,688 high-quality stories across 28 languages. Each… See the full description on the dataset page: https://huggingface.co/datasets/Dxniz/TinyStories-Multilingual.multilingual-sentences
Multilingual Sentences
Dataset contains sentences from 50 languages, grouped by their two-letter ISO 639-1 codes. The "all" configuration includes sentences from all languages.
Dataset Overview
Multilingual Sentence Dataset is a comprehensive collection of high-quality, linguistically diverse sentences. Dataset is designed to support a wide range of natural language processing tasks, including but not limited to language modeling, machine translation, and cross-lingual… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/multilingual-sentences.emojinize-multilingual
Emojinize Multilingual
A multilingual dataset of 108,478 sentences across 14 languages for emoji-based text augmentation. In each sentence, selected spans (individual words or fixed multi-word expressions) are identified by character offsets and paired with emoji sequences representing their meaning in context. The dataset supports downstream span detection and emoji generation tasks, and was created using a two-stage LLM annotation pipeline with gpt-5.4 for span marking and… See the full description on the dataset page: https://huggingface.co/datasets/yagizgencer/emojinize-multilingual.multilingual_tokenizer_benchmark
Multilingual Tokenizer Benchmark
More details of each subset like word count, character count, original sources, etc, can be found in the dataset_meta.yaml file in the repository root.
Natural language word count functions
Download spacy models
pip install ntlk spacy pygments underthesea camel-tools
python -m spacy download ko_core_news_sm
python -m spacy download ja_core_news_sm
python -m spacy download zh_core_web_sm
import nltk
nltk.download('punkt_tab')… See the full description on the dataset page: https://huggingface.co/datasets/eduagarcia/multilingual_tokenizer_benchmark.multilingual_paradetoxMultilingual Text Detoxification with Parallel Data
This is the multilingual parallel dataset for the text detoxification task. Prepared for TextDetox Shared Task.
📰 Updates
[2025] The second edition of TextDetox shared task! webpage
[2025] We extend our data to new languages! Now also included: Italian, French, Hebrew, Hinglish, Japanese, Tatar. Check our test part.
[2025]We dived into the explainability of our data in our new COLING paper!
[2024] You can check additional releases for… See the full description on the dataset page: https://huggingface.co/datasets/textdetox/multilingual_paradetox.tiny-aya-l2-thinker-multilingual-reasoning
Tiny Aya L2 Multilingual Reasoning (44 languages)
Translated multilingual reasoning traces used to train Tiny Aya L2-Thinker.
Each example has the prompt, thinking, and answer in the same non-English language alongside the original texts in English.
Languages (44)
Language
Train
Test
Total
Amharic (am)
3,807
448
4,255
Arabic (ar)
22,968
2,538
25,506
Bulgarian (bg)
4,177
452
4,629
Bengali (bn)
3,803
422
4,225
Catalan (ca)
4,251
512
4,763
Czech… See the full description on the dataset page: https://huggingface.co/datasets/erenyeager-1/tiny-aya-l2-thinker-multilingual-reasoning.multilingual-textarena-ColonelBlotto-v0-train
TextArena Language Trajectories
This dataset contains language-conditioned TextArena trajectory data.
Each dataset configuration corresponds to a different model, experiment group, or source folder.
Available configurations:
gemma4-e4b-it
qwen3-4b
ministral3-3b-instruct
Usage
Install the datasets library:
pip install datasets
Load a specific configuration:
from datasets import load_dataset
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/The-CoLab/multilingual-textarena-ColonelBlotto-v0-train.novel-multilingual
WebNovel Multilingual Dataset
Dataset Description
This dataset contains web novels scraped from WebNovel.com across multiple languages. Each entry includes the complete novel content with chapter information, metadata, and classification tags.
Note: This dataset excludes content in the following languages: id, ID
Dataset Statistics
Total Novels: 8,324
Total Chapters: 233,410
Total Characters: 1,617,589,129
Languages: 10
Language Distribution… See the full description on the dataset page: https://huggingface.co/datasets/taozi555/novel-multilingual.