datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Vript_Multilingual
🎬 Vript: A Video Is Worth Thousands of Words [Github Repo]
We construct another fine-grained video-text dataset with 19.1K annotated high-resolution UGC videos (~677k clips) in multiple languages to be the Vript_Multilingual.
New in Vript_Multilingual:
Multilingual: zh (60%), en (17%), de (15%), ja (6%), ko (2%), ru (<1%), es (<1%), pt (<1%), jv (<1%), fr (<1%), id (<1%), vi (<1%)
More diverse and fine-grained categories: 113 categories (please check vript_CN-V2_meta.json)… See the full description on the dataset page: https://huggingface.co/datasets/Mutonix/Vript_Multilingual.McEvalMcEval benchmark data as described in the McEval Paper. Code for the evaluation can be found on Github as McEval.
IfEvalCode-testsetmultilingual_mbppMBPP translated to 15 programming languages using o4-mini-medium.
source_language = "python"
target_languages = [
"cpp",
"c",
"javascript",
"java",
"php",
"csharp",
"typescript",
"bash",
"swift",
"go",
"rust",
"ruby",
"r",
"matlab",
"scala",
"haskell"
]
effort = "medium"
dataset_name = "google-research-datasets/mbpp"
model = "o4-mini"
Nemotron-SFT-Multilingual-v2
Dataset Description:
Nemotron-SFT-Multilingual-v2 is a multilingual supervised fine-tuning (SFT) dataset for post-training text-generation models. It is generated by translating seed data from Nemotron-Math-v2, Nemotron-Competitive-Programming-v1, and Nemotron-Science-v1, adding multilingual coverage for Hindi (hi), Korean (ko), Brazilian Portuguese (pt-br), and refreshed Japanese (ja) data.
The dataset is generated with a new data processing pipeline that avoids line-breaking… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-SFT-Multilingual-v2.multilingual-sycophancy
Multilingual Sycophancy
A Parallel Benchmark for Cross-Lingual Alignment Failure across 38 Languages, 33 Opinion Categories, and 3 Resource Tiers.
This dataset accompanies the research paper Sycophancy as a Multilingual Alignment Failure: How Safety Degrades Across Languages, Topics, and Models. It contains 188,100 parallel records (4,950 per language × 38 languages) — each a triple of (prompt, sycophantic response, non-sycophantic response) — designed for forced-choice… See the full description on the dataset page: https://huggingface.co/datasets/aryashah00/multilingual-sycophancy.gsm8k-multilingual-reasoning
gsm8k-multilingual-reasoning
GSM8K with reasoning translated to multiple languages
Schema
{"prompt": "...", "answer": "...", "reasoning": "...", "metadata": {...}}
Usage
from datasets importload_dataset
ds = load_dataset("eddie-OB/gsm8k-multilingual-reasoning")
print(ds["train"][0])
Source
Derived from OpenAI GSM8K.
multilingual-phonemes-10k-alpha
Multilingual Phonemes 10K Alpha
This dataset contains approximately 10,000 pairs of text and phonemes from each supported language. We support 15 languages in this dataset, so we have a total of ~150K pairs. This does not include the English-XL dataset, which includes another 100K unique rows.
Languages
We support 15 languages, which means we have around 150,000 pairs of text and phonemes in multiple languages. This excludes the English-XL dataset, which has 100K unique… See the full description on the dataset page: https://huggingface.co/datasets/styletts2-community/multilingual-phonemes-10k-alpha.IfEvalCode-InstructMdEval
MDEVAL: Massively Multilingual Code Debugging
Official repository for our paper "MDEVAL: Massively Multilingual Code Debugging"
🏠 Home Page •
📊 Benchmark Data •
🏆 Leaderboard
Introduction
MDEVAL is a massively multilingual debugging benchmark covering 20 programming languages with 3.9K test samples and three tasks focused on bug fixing. It substantially pushes the limits of code LLMs in multilingual… See the full description on the dataset page: https://huggingface.co/datasets/Multilingual-Multimodal-NLP/MdEval.gsm8k-multilingual
gsm8k-multilingual
GSM8K translated to multiple languages (no reasoning)
Schema
{"prompt": "...", "answer": "...", "metadata": {...}}
Usage
from datasets import load_dataset
ds = load_dataset("eddie-OB/gsm8k-multilingual")
print(ds["train"][0])
Source
Derived from OpenAI GSM8K.
high-quality-multilingual-sentences
High Quality Multilingual Sentences
This dataset contains multilingual sentences derived from the agentlans/LinguaNova dataset.
It includes 1.58 million rows across 51 different languages, each in its own configuration.
Example row (from the all config):
{
"text": "امام جمعه اصفهان گفت: میزان نیاز آب شرب اصفهان ۱۱.۵ متر مکعب است که تمام استان اصفهان را پوشش میدهد و نسبت به قبل از انقلاب یکی از پیشرفتها در حوزه آب بوده است.",
"fasttext": "fa",
"gcld3": "fa"
}
Fields:… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/high-quality-multilingual-sentences.cs_csfd-movie-reviews
Dataset Card for CSFD movie reviews (Czech)
Dataset Description
The dataset contains user reviews from Czech/Slovak movie databse website https://csfd.cz.
Each review contains text, rating, date, and basic information about the movie (or TV series).
The dataset has in total (train+validation+test) 30,000 reviews. The data is balanced - each rating has approximately the same frequency.
Dataset Features
Each sample contains:
review_id: unique string identifier… See the full description on the dataset page: https://huggingface.co/datasets/fewshot-goes-multilingual/cs_csfd-movie-reviews.pile
The Pile Replication Code
The official website for the the Pile is here.
The Pile is a large, diverse, open source language modelling data set that consists of many smaller datasets combined together. The objective is to obtain text from as many modalities as possible to ensure that models trained using The Pile will have much broader generalization abilities.
This repository is for replicating or making variants of the Pile. IF YOU ARE HERE TO USE THE PILE DATASET, THIS… See the full description on the dataset page: https://huggingface.co/datasets/multilingual-mi-llm/pile.nym-pii-multilingual-data
nym-pii-multilingual-data
805,000 synthetic, exactly-labeled PII token-classification examples across
~23 languages and 6 scripts, built for training
nym's PII detection models
(e.g. Wismut/nym-pii-multilingual).
Format
JSONL with character-offset spans (offsets index into text as UTF-8 —
compatible with HF fast-tokenizer offset_mapping):
{"text": "Passport Y94316756 issued to Gary Fisher, Ukraine, expires 11/03/2008.",
"entities": [{"start": 9, "end": 18… See the full description on the dataset page: https://huggingface.co/datasets/Wismut/nym-pii-multilingual-data.IndicNLP-Multilingualrisale-nur-multilingual
Risale-i Nur Multilingual Corpus
Bediüzzaman Said Nursî'nin Risale-i Nur külliyatının 27 dilde çok dilli korpusu — her eser başlıklara göre bölümlere (section) ayrılmış, bölümler diller arasında hizalanmış ve konu (topic) hiyerarşisiyle etiketlenmiştir.
Güncel release: v2.10.0 · 20 config/lane · 163,820 config-split satırı. Alt başlıklardaki eski v2.x etiketleri lane'in ilk eklendiği sürümü gösterir; güncel release sürümü değildir. Deterministik projeksiyonlar duplicate_of ile… See the full description on the dataset page: https://huggingface.co/datasets/risaleinur/risale-nur-multilingual.TinyStories-Multilingual
Novelist: TinyStories Multilingual Edition
Dataset Summary
The TinyStories Multilingual Edition is a high-fidelity synthetic dataset of short, child-safe fiction designed to stress-test literary consistency, emotional warmth, and multilingual fluency in small models. Derived from the broader Novelist ecosystem, this subset focuses on narrative simplicity paired with complex moral and social themes.
The dataset contains 15,688 high-quality stories across 28 languages. Each… See the full description on the dataset page: https://huggingface.co/datasets/Dxniz/TinyStories-Multilingual.multilingual-queries-2026
MAST Multilingual Queries 2026
This dataset contains the multilingual query set for MAST @ FIRE 2026, the Multilingual Agentic Search Track. MAST evaluates whether multilingual agentic search systems can answer complex questions posed in different languages.
MAST builds on BrowseComp-Plus (ACL 2026), a reproducible and verifiable extension of BrowseComp with challenging English queries, a verified English corpus of roughly 100K web-sourced documents, and human judgments. In the… See the full description on the dataset page: https://huggingface.co/datasets/mast-benchmark/multilingual-queries-2026.multilingual-safety-classification-dataset
Multilingual Safety Classification Dataset
A multilingual dataset for safety classification across 60 languages, created by Hasan Kurşun through machine translation of English safety prompts using NLLB-200-3.3B.
Dataset Details
Processed by: Hasan KurşunAuthor: Hasan KurşunYear: 2025Source Dataset: mvrcii/safety-moderation-benchmarkTranslation Model: facebook/nllb-200-3.3B
Languages (60)
African Languages (16): Amharic, Hausa, Kinyarwanda, Luganda… See the full description on the dataset page: https://huggingface.co/datasets/hasankursun/multilingual-safety-classification-dataset.McEval-InstructMcEval-Instruct data as described in the McEval Paper. Code for the evaluation and sft can be found on Github as McEval.
bge-multilingual-gemma2-data
Dataset Summary
Training Data of bge-multilingual-gemma2 (For the details of each dataset, please refer to Appendix C of the paper: https://arxiv.org/pdf/2409.15700):
English:
ArguAna:
config_name: en_arguana
available splits: train
COLIEE
config_name: en_coliee
available splits: train
ELI5
config_name: en_eli5
available splits: train
en_NLI_data
config_name: en_nli_data
available splits: train
en_STS_data
config_name: en_sts_data
available splits: train… See the full description on the dataset page: https://huggingface.co/datasets/hanhainebula/bge-multilingual-gemma2-data.TableInstruct
Citation
@misc{wu2024tablebenchcomprehensivecomplexbenchmark,
title={TableBench: A Comprehensive and Complex Benchmark for Table Question Answering},
author={Xianjie Wu and Jian Yang and Linzheng Chai and Ge Zhang and Jiaheng Liu and Xinrun Du and Di Liang and Daixin Shu and Xianfu Cheng and Tianzhen Sun and Guanglin Niu and Tongliang Li and Zhoujun Li},
year={2024},
eprint={2408.09174},
archivePrefix={arXiv},
primaryClass={cs.CL}… See the full description on the dataset page: https://huggingface.co/datasets/Multilingual-Multimodal-NLP/TableInstruct.multilingual_tokenizer_benchmark
Multilingual Tokenizer Benchmark
More details of each subset like word count, character count, original sources, etc, can be found in the dataset_meta.yaml file in the repository root.
Natural language word count functions
Download spacy models
pip install ntlk spacy pygments underthesea camel-tools
python -m spacy download ko_core_news_sm
python -m spacy download ja_core_news_sm
python -m spacy download zh_core_web_sm
import nltk
nltk.download('punkt_tab')… See the full description on the dataset page: https://huggingface.co/datasets/eduagarcia/multilingual_tokenizer_benchmark.Multilingual-Normalizer
Multilingual TTS text normalizer (written → spoken)
Training pairs for fine-tuning a small LLM as a text-to-speech normalizer: text is a sentence
the way people type it (digits, currency symbols, dates, phone numbers, …) and normalized is the
exact spoken form, in the same language, with nothing left that a TTS model cannot say.
52,698 rows — 16 monolingual locales and 6 Malaysian code-switched pairs. Every row is
digit-free on the spoken side.
from datasets import load_dataset… See the full description on the dataset page: https://huggingface.co/datasets/Scicom-intl/Multilingual-Normalizer.low-quality-multilingual-sentences
Low Quality Multilingual Sentences
This dataset is a complement to agentlans/high-quality-multilingual-sentences to extend it to more languages.
The new sentences in this dataset are low quality, proceed with caution.
global-multilingual-cross-industry-corpus
Global Multilingual Cross-Industry Corpus
A highly structured, clean, and comprehensive cross-industry text corpus compiled across specialized enterprise domains, covering multiple manufacturing sectors and global languages. This dataset is explicitly optimized for Vertical Industry LLM Fine-tuning, Multi-lingual Machine Translation (MT), Domain-Specific RAG (Retrieval-Augmented Generation) systems, and AI crawler evaluation.
📊 Dataset Overview
Unlike mixed… See the full description on the dataset page: https://huggingface.co/datasets/Susan2026/global-multilingual-cross-industry-corpus.AutoMemoryBench
AutoMemoryBench
State-Contract Evaluation for Auditable Agent Memory
AutoMemoryBench evaluates whether an agent uses the right memory, and only
the admissible memory, under a query-time state contract. Each executable
contract partitions memory into required, admissible, and
prohibited sets. Prohibited memories are typed as superseded, deleted,
restricted, cross-namespace, or stale-tool.
Relevance is not enough: remembered evidence must also be… See the full description on the dataset page: https://huggingface.co/datasets/Multilingual-Multimodal-NLP/AutoMemoryBench.cs_czech-named-entity-corpus_2.0
Dataset Card for Czech Named Entity Corpus 2.0
Dataset Description
The dataset contains Czech sentences and annotated named entities. Total number of sentences is around 9,000 and total number of entities is around 34,000. (Total means train + validation + test)
Dataset Features
Each sample contains:
text: source sentence
entities: list of selected entities. Each entity contains:
category_id: string identifier of the entity category
category_str:… See the full description on the dataset page: https://huggingface.co/datasets/fewshot-goes-multilingual/cs_czech-named-entity-corpus_2.0.agri-vet-multilingual-dataset
Agri-Vet Multilingual Dataset
This repository contains JSON and JSONL resources for multilingual agricultural and veterinary language tasks. It is intended to support conversational, retrieval, classification, or instruction-tuning experiments spanning crop, animal, and related user questions.
Working with the files
Inspect each JSON/JSONL record and preserve its language, domain, prompt, response, label, and provenance fields when creating a derived dataset.… See the full description on the dataset page: https://huggingface.co/datasets/MR-CODESPIKE/agri-vet-multilingual-dataset.