datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
McEvalMcEval benchmark data as described in the McEval Paper. Code for the evaluation can be found on Github as McEval.
Nemotron-SFT-Multilingual-v2
Dataset Description:
Nemotron-SFT-Multilingual-v2 is a multilingual supervised fine-tuning (SFT) dataset for post-training text-generation models. It is generated by translating seed data from Nemotron-Math-v2, Nemotron-Competitive-Programming-v1, and Nemotron-Science-v1, adding multilingual coverage for Hindi (hi), Korean (ko), Brazilian Portuguese (pt-br), and refreshed Japanese (ja) data.
The dataset is generated with a new data processing pipeline that avoids line-breaking… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-SFT-Multilingual-v2.WangchanThaiInstruct_Multi-turn_Conversation_Dataset
WangchanThaiInstruct Multi-turn Conversation Dataset
We create a Thai multi-turn conversation dataset from airesearch/WangchanThaiInstruct (Batch 1) by LLM. It was created from synthetic method using open source LLM in Thai language.
Citation
Thammaleelakul, S., & Phatthiyaphaibun, W. (2024). WangchanThaiInstruct Multi-turn Conversation Dataset [Data set]. Zenodo. https://doi.org/10.5281/zenodo.13132633
or BibTeX
@dataset{thammaleelakul_2024_13132633,
author =… See the full description on the dataset page: https://huggingface.co/datasets/ThaiSyntheticQA/WangchanThaiInstruct_Multi-turn_Conversation_Dataset.multiloko
MultiLoKo: a multilingual local knowledge benchmark for LLMs
MultiLoKo is a multilingual knowledge benchmark, covering 30 languages plus English.
The questions are separately sourced for each language, with an annotation protocol designed to target locally relevant topics for the respective language.
MultiLoKo contains the original data for each language, as well as both human and machine-authored translations of each non-English subset into English and vice versa, facilitating… See the full description on the dataset page: https://huggingface.co/datasets/facebook/multiloko.DEBATE
DEBATE: Diverse Multi-Agent Debates
This dataset is presented in the paper "MALLM: Multi-Agent Large Language Models Framework".
Citation
comming soon.
gsm8k-multilingual-reasoning
gsm8k-multilingual-reasoning
GSM8K with reasoning translated to multiple languages
Schema
{"prompt": "...", "answer": "...", "reasoning": "...", "metadata": {...}}
Usage
from datasets importload_dataset
ds = load_dataset("eddie-OB/gsm8k-multilingual-reasoning")
print(ds["train"][0])
Source
Derived from OpenAI GSM8K.
Multi-turn_Long-context_Benchmark_for_LLMs
LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues
Arxiv: https://www.arxiv.org/abs/2507.13681
Huggingface: https://huggingface.co/papers/2507.13681
Introduction
LoopServe Multi-Turn Dialogue Benchmark is a comprehensive evaluation dataset comprising multiple diverse datasets designed to assess large language model performance in realistic conversational scenarios.
Unlike traditional benchmarks that place queries only at the end… See the full description on the dataset page: https://huggingface.co/datasets/TreeAILab/Multi-turn_Long-context_Benchmark_for_LLMs.gsm8k-multilingual
gsm8k-multilingual
GSM8K translated to multiple languages (no reasoning)
Schema
{"prompt": "...", "answer": "...", "metadata": {...}}
Usage
from datasets import load_dataset
ds = load_dataset("eddie-OB/gsm8k-multilingual")
print(ds["train"][0])
Source
Derived from OpenAI GSM8K.
Nemotron-RL-Multichallenge-v1Nemotron-RL-Multichallenge-v1
License: cc-by-4.0
Task Categories: reinforcement-learning, text-generation
Tags: instruction-following, multi-turn, rlvr, nemo-gym
Configs: advanced train split at data/advanced.jsonl; vanilla train split at data/vanilla.jsonl
Domain: instruction following, multi-turn conversation
Modality: text
Capability Breakdown: Multi-turn instruction following [100%]
Source: Hybrid: Manually Collected, Synthetic
Size Bin: <10K
Associated Model Release: Nemotron Ultra… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-RL-Multichallenge-v1.multilingual-sycophancy
Multilingual Sycophancy
A Parallel Benchmark for Cross-Lingual Alignment Failure across 38 Languages, 33 Opinion Categories, and 3 Resource Tiers.
This dataset accompanies the research paper Sycophancy as a Multilingual Alignment Failure: How Safety Degrades Across Languages, Topics, and Models. It contains 188,100 parallel records (4,950 per language × 38 languages) — each a triple of (prompt, sycophantic response, non-sycophantic response) — designed for forced-choice… See the full description on the dataset page: https://huggingface.co/datasets/aryashah00/multilingual-sycophancy.McEval-InstructMcEval-Instruct data as described in the McEval Paper. Code for the evaluation and sft can be found on Github as McEval.
TinyStories-Multilingual
Novelist: TinyStories Multilingual Edition
Dataset Summary
The TinyStories Multilingual Edition is a high-fidelity synthetic dataset of short, child-safe fiction designed to stress-test literary consistency, emotional warmth, and multilingual fluency in small models. Derived from the broader Novelist ecosystem, this subset focuses on narrative simplicity paired with complex moral and social themes.
The dataset contains 15,688 high-quality stories across 28 languages. Each… See the full description on the dataset page: https://huggingface.co/datasets/Dxniz/TinyStories-Multilingual.high-quality-multilingual-sentences
High Quality Multilingual Sentences
This dataset contains multilingual sentences derived from the agentlans/LinguaNova dataset.
It includes 1.58 million rows across 51 different languages, each in its own configuration.
Example row (from the all config):
{
"text": "امام جمعه اصفهان گفت: میزان نیاز آب شرب اصفهان ۱۱.۵ متر مکعب است که تمام استان اصفهان را پوشش میدهد و نسبت به قبل از انقلاب یکی از پیشرفتها در حوزه آب بوده است.",
"fasttext": "fa",
"gcld3": "fa"
}
Fields:… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/high-quality-multilingual-sentences.risale-nur-grounded-multipool
Risale-i Nur Grounded Multi-Pool LLM Dataset
TR. 15 kanonik Risale-i Nur kitabından hazırlanan; kaynak
bağlı üretim, SFT, tercih, değerlendirme, sürekli ön eğitim ve erişim
çalışmaları için çok görünümlü bir veri seti.
EN. A multi-view dataset built from 15 canonical Risale-i
Nur books for grounded generation, SFT, preference learning, evaluation,
continued pretraining, and retrieval.
v2.10.0 · 199 configs · 463 config/split views ·
527,196 rows across configured views… See the full description on the dataset page: https://huggingface.co/datasets/risaleinur/risale-nur-grounded-multipool.risale-nur-multilingual
Risale-i Nur Multilingual Corpus
Bediüzzaman Said Nursî'nin Risale-i Nur külliyatının 27 dilde çok dilli korpusu — her eser başlıklara göre bölümlere (section) ayrılmış, bölümler diller arasında hizalanmış ve konu (topic) hiyerarşisiyle etiketlenmiştir.
Güncel release: v2.10.0 · 20 config/lane · 163,820 config-split satırı. Alt başlıklardaki eski v2.x etiketleri lane'in ilk eklendiği sürümü gösterir; güncel release sürümü değildir. Deterministik projeksiyonlar duplicate_of ile… See the full description on the dataset page: https://huggingface.co/datasets/risaleinur/risale-nur-multilingual.multiturn-chatMultiAgentFraudBench
MultiAgentFraudBench Dataset
中文 | English
🌐 Project Page
| 📄 Paper
| 📦 Code
This directory contains the MultiAgentFraudBench dataset, a comprehensive collection of synthetic financial fraud posts designed for multi-agent fraud simulation research. The dataset is generated through a multi-agent simulation framework built on OASIS, capturing realistic fraud lifecycle from initial posts, trust-building through collusion, to victim-fraudster dialogues. All content… See the full description on the dataset page: https://huggingface.co/datasets/ninty-seven/MultiAgentFraudBench.multilingual_tokenizer_benchmark
Multilingual Tokenizer Benchmark
More details of each subset like word count, character count, original sources, etc, can be found in the dataset_meta.yaml file in the repository root.
Natural language word count functions
Download spacy models
pip install ntlk spacy pygments underthesea camel-tools
python -m spacy download ko_core_news_sm
python -m spacy download ja_core_news_sm
python -m spacy download zh_core_web_sm
import nltk
nltk.download('punkt_tab')… See the full description on the dataset page: https://huggingface.co/datasets/eduagarcia/multilingual_tokenizer_benchmark.Multilingual-Normalizer
Multilingual TTS text normalizer (written → spoken)
Training pairs for fine-tuning a small LLM as a text-to-speech normalizer: text is a sentence
the way people type it (digits, currency symbols, dates, phone numbers, …) and normalized is the
exact spoken form, in the same language, with nothing left that a TTS model cannot say.
52,698 rows — 16 monolingual locales and 6 Malaysian code-switched pairs. Every row is
digit-free on the spoken side.
from datasets import load_dataset… See the full description on the dataset page: https://huggingface.co/datasets/Scicom-intl/Multilingual-Normalizer.RAG_Multilingual
Dataset Card for RAG_Multilingual
Dataset Summary
RAG_Multilingual is an instruction-following synthetic QA dataset created from extractive QA datasets from Catalan, English and Spanish reference sets.
The reference datasets were: SQAD (https://huggingface.co/datasets/rajpurkar/squad), Catalanqa (https://huggingface.co/datasets/projecte-aina/catalanqa) and SQAC (https://huggingface.co/datasets/PlanTL-GOB-ES/SQAC).
This dataset, of 56.406 instances, was created by… See the full description on the dataset page: https://huggingface.co/datasets/projecte-aina/RAG_Multilingual.All-CVE-Chat-MultiTurn-1999-2025-Dataset
CVE Chat‑Style Multi‑Turn Cybersecurity Dataset (1999 – 2025)
1. Project Overview
This repository hosts the largest publicly available chat‑style, multi‑turn cybersecurity dataset to date, containing ≈ 300 000 Common Vulnerabilities and Exposures (CVE) records published between 1999 and 2025. Each record has been meticulously parsed, enriched, and converted into a conversational format that is ideal for training and evaluating AI and AI‑Agent systems focused on… See the full description on the dataset page: https://huggingface.co/datasets/Trendyol/All-CVE-Chat-MultiTurn-1999-2025-Dataset.multilingual-mathematical-autoformalization
Multilingual Mathematical Autoformalization
"Paper"
This repository contains parallel mathematical statements:
Input: An informal proof in natural language
Output: The corresponding formalization in either Lean or Isabelle
This dataset can be used to train models how to formalize mathematical statements into verifiable proofs, a form of machine translation.
Abstract
Autoformalization is the task of translating natural language materials into machine-verifiable… See the full description on the dataset page: https://huggingface.co/datasets/casey-martin/multilingual-mathematical-autoformalization.BenchMAX_Multiple_Functions
Dataset Sources
Paper: BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models
Link: https://huggingface.co/papers/2502.07346
Repository: https://github.com/CONE-MT/BenchMAX
Dataset Description
BenchMAX_Multiple_Functions is a dataset of BenchMAX, sourcing from Nexus.
This dataset evaluates the tool use capability in multilingual senarios, which requires a model to call the correct function given the user query and multiple functions.
We… See the full description on the dataset page: https://huggingface.co/datasets/LLaMAX/BenchMAX_Multiple_Functions.turkish-qa-multi-dialog-dataset
Turkish QA & Multi-Dialog Dataset
Bu depo, iki farklı Türkçe veri kaynağının birleştirilmiş ve temizlenmiş sürümünü içerir:
Yaklaşık 19.000 adet soru-cevap (QA) örneği
Çok adımlı, doğal Türkçe sohbetlerden oluşan diyalog verileri
Bu dataset, hem genel amaçlı Türkçe QA modelleri hem de sohbet/chatbot modelleri için uygundur.
Veri İçeriği
QA Bölümü (~19K)
SQuAD benzeri yapıdan dönüştürülmüş input–output örnekleri
Her satır: tek bir soru ve net bir cevap… See the full description on the dataset page: https://huggingface.co/datasets/sixfingerdev/turkish-qa-multi-dialog-dataset.Dendrite-Synth-Multi-Domain
Dendrite Synth Multi-Domain
A verified, difficulty-filtered, style-amplified synthetic corpus of question / reasoning / answer
triples spanning the 14 MMLU-Pro categories - mathematics, computer science, natural sciences, chemistry, physics, engineering, health, law, business, economics, psychology, philosophy, history and others (expanded to 122 fine-grained categories and 695 subcategories). Problems are written by a pool of generator models, solved with explicit reasoning by… See the full description on the dataset page: https://huggingface.co/datasets/dendriteholdings/Dendrite-Synth-Multi-Domain.multimodal-video-annotation-samples
Video Annotation Samples – SuperviseLab
SuperviseLab provides professional video annotation data for training multimodal AI models. This public sample dataset demonstrates our annotation methodology and output quality across diverse video content categories.
Note: All visual assets in this dataset have been abstracted (pixelated mosaic) to protect source privacy. Uploader identity, original titles, and all identifiable metadata have been removed. This is a demonstration dataset… See the full description on the dataset page: https://huggingface.co/datasets/superviselab/multimodal-video-annotation-samples.low-quality-multilingual-sentences
Low Quality Multilingual Sentences
This dataset is a complement to agentlans/high-quality-multilingual-sentences to extend it to more languages.
The new sentences in this dataset are low quality, proceed with caution.
Multilingal-sakalt-dataマルチリンガルデータセットです。mitライセンスです。
MultiAgent-X
MultiAgent-X: Multilingual Agentic Function-Calling Benchmark
Created with Adaptive Data by Adaption
The first open-source multilingual function-calling training and evaluation dataset targeting under-resourced languages. 10,551 records across 12 languages, 7 unique writing systems, and 5 life-critical agentic domains covering 1.3 billion speakers that mainstream AI has never been optimised for.
The Gap This Fills
MASSIVE-Agents (EMNLP 2025) evaluated multilingual… See the full description on the dataset page: https://huggingface.co/datasets/Saurabh-66/MultiAgent-X.qa-expert-multi-hop-qa-V1.0
Dataset Card for QA-Expert-multi-hop-qa-V1.0
This dataset aims to provide multi-domain training data for the task: Question Answering, with a focus on Multi-hop Question Answering.
In total, this dataset contains 25.5k for training and 3.19k for evaluation.
You can take a look at the model we trained on this data: https://huggingface.co/khaimaitien/qa-expert-7B-V1.0
The dataset is mostly generated using the OpenAPI model (gpt-3.5-turbo-instruct). Please read more information about… See the full description on the dataset page: https://huggingface.co/datasets/khaimaitien/qa-expert-multi-hop-qa-V1.0.