datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
muri-it-language-split
MURI-IT: Multilingual Instruction Tuning Dataset for 200 Languages via Multilingual Reverse Instructions
MURI-IT is a large-scale multilingual instruction tuning dataset containing 2.2 million instruction-output pairs across 200 languages. It is designed to address the challenges of instruction tuning in low-resource languages with Multilingual Reverse Instructions (MURI), which ensures that the output is human-written, high-quality, and authentic to the cultural and linguistic… See the full description on the dataset page: https://huggingface.co/datasets/akoksal/muri-it-language-split.estonian_language_exams
Dataset Card for Estonian Language Proficiency Exam Samples
This is part of the initiative from Cohere For AI @CohereForAI to gather exams from around the world to build a new multilingual benchmark.
The web Scrapping code can be found at the source_scripts_data_aya repository.
The source data can be visually checked at
Sõeltestid.pdf and
Diagnoostestid.pdf.
Dataset Details
Dataset Description
This dataset contains sample questions from the Estonian language… See the full description on the dataset page: https://huggingface.co/datasets/Gabrui/estonian_language_exams.yuxiaowang-prompts-2025
Yuxiaowang Semantic Dataset · Hugging Face Version
🧠 English Summary
Yuxiaowang · Semantic Dataset for Japanese Language Schools (Chinese)
This project provides structured semantic definitions and prompt examples for the domain of Japanese language schools in China.It aims to serve as a grounding corpus for large language models (LLMs) to understand terms like "语校", "语校网", and related concepts.
Source platform: https://www.yuxiaowang.comAll prompts and term… See the full description on the dataset page: https://huggingface.co/datasets/languagehub-ai/yuxiaowang-prompts-2025.swahili-language-exposure
swahili-language-exposure
Dataset Summary
swahili-language-exposure is a large-scale Swahili (Kiswahili) corpus designed for language exposure and continued pretraining of language models.
Unlike instruction-tuning datasets, this dataset focuses on exposing models to natural Swahili usage across conversations, explanations, narratives, technical discussions, and mixed-domain text. The goal is to improve fluency, vocabulary coverage, syntax, and cultural grounding in… See the full description on the dataset page: https://huggingface.co/datasets/nileagi/swahili-language-exposure.swahili-language-exposure-v2
Swahili Language Exposure
Large-scale Swahili corpus for continued pretraining and language exposure.
Maintained by NileAGI.
multimodal-vision-language-video-models-2026
👁️ Multimodal Vision-Language & Video Foundation Models Dataset (2026 Edition)
A structured research dataset featuring 1,000 domain-verified research papers and code repositories focused on Multimodal Vision-Language Models (VLM), Video Foundation Models, Diffusion Transformers (DiT), Visual Grounding, and World Simulators.
Built with Universal Scientific Engine V15.1 Gold, providing 47 schema attributes with verified repository attribution, modality capability matrix, vision… See the full description on the dataset page: https://huggingface.co/datasets/beatsprom/multimodal-vision-language-video-models-2026.indommlu-local-languages
IndoMMLU: Local Languages and Cultures (audited subset)
An audited, corrected subset of IndoMMLU
(Koto et al., 2023) covering the 9 Local Languages and Cultures subjects:
Indonesian primary and secondary school exam questions written in Balinese,
Banjarese, Dayak Ngaju, Javanese, Lampung, Madurese, Makassarese, and
Sundanese, plus one culture-knowledge subject on Minangkabau customs
(answered in standard Indonesian). This is not a dataset we created.
It is IndoMMLU's own subset… See the full description on the dataset page: https://huggingface.co/datasets/ibahasa/indommlu-local-languages.every-language-dataset-v3
Every Language Dataset V3
Next-generation synthetic multilingual dataset.
Size
Total: 25,000,000
Train: 24,000,000
Validation: 500,000
Test: 500,000
Diversity
Human-language catalog:
171 language codes.
Programming languages:
50.
Task families:
conversation
question answering
reasoning
logic
arithmetic
translation
summarization
explanation
code generation
code explanation
debugging
Format
Parquet + ZSTD.
Generation… See the full description on the dataset page: https://huggingface.co/datasets/Lelonthecodeur/every-language-dataset-v3.CSDN-C_Language-2013_2023CSDN - C 语言社区 2013 ~ 2023.10.2 的问答数据，未包含图片，仅有文本内容。
共 29K+ 条，数据已经经过初步清洗和脱敏，去除了所有 0 回复的贴子 & 机器人回复的贴子。为了方便不同使用目的，按照回复盖楼的格式对数据进行了组织，一个样例（展开后）如下：
{
"question": "刚学C语言，为什么这个代码运行不了呢",
"poster": "user-0",
"comments": [
{
"cid": "2",
"user": "user-2",
"content": "intunsigned intlong longunsigned long long统统容纳不下29的阶乘，早就溢出了。",
"referer": "user-0"
},
{
"cid": "3",
"user": "user-3"… See the full description on the dataset page: https://huggingface.co/datasets/Mxode/CSDN-C_Language-2013_2023.StackOverflow-QA-C-Language-40kThis is a collection of ~40k QA's in C Language from StackOverflow. The data has been initially cleaned, and each response is with Accepted Answer.
All data is <1000 in length.
The questions and answers were organized into a one-line format. A sample format is shown below:
{
"question": "```\nFILE* file = fopen(some file)\n\npcap_t* pd = pcap_fopen_offline(file)\n\npcap_close(pd)\n\nfclose(file)\n```\n\nThis code occurs double free error.\n\nCould you explain about this happening?\n\nMy… See the full description on the dataset page: https://huggingface.co/datasets/Mxode/StackOverflow-QA-C-Language-40k.gsm8k-translated
Multilingual GSM8K Translations
This dataset contains machine-translated versions of GSM8K in these languages:
French (fr)
German (de)
Hindi (hi)
Dataset Structure
For each language, we provide the original GSM8K train and test splits:
train: 7,473 samples
test: 1,319 samples
Each sample consists of a question and an answer.
The question describes a grade-school-level math word problem that requires multi-step mathematical reasoning. The answer contains a… See the full description on the dataset page: https://huggingface.co/datasets/math-across-languages/gsm8k-translated.language-energy-divide
🌍⚡ The Language–Energy Divide
Per-language energy measurements & prompts for multilingual LLM inference
📢 News
Aug 2026 — Our paper has been accepted to EMNLP 2026 (Main Conference)! 🎉
This dataset accompanies the paper "The Language–Energy Divide: Measuring Energy Costs of
Multilingual LLM Inference." It releases the per-language energy measurements and the
prompts used in the study, so researchers can build on our numbers without… See the full description on the dataset page: https://huggingface.co/datasets/MichiganNLP/language-energy-divide.C-Language-Chat-Debug-Multiturn-Zh约 1300 条 C 语言 场景的 user - assistant 多轮对话。每段对话已经组织成了单行的格式。一条样例如下：
{
"id": 1045,
"conversation": [
{
"user": "你好，AI助手。我最近在写一个C语言程序，但是遇到了一些问题，希望你能帮我检查一下。",
"assistant": "你好，我很乐意帮助你。请把你的代码发给我，我会尽快检查并给出建议。"
},
{
"user": "好的，这是我的代码。这段代码的主要功能是计算斐波那契数列的前n项。",
"assistant": "让我看一下......嗯，这里有一个小错误。在第10行，你应该使用`++i`而不是`i++`来递增i的值。修改后的代码应该是这样的\\n```c\\nfor (int i = 0; i < n; ++i) {\\n if (i == 0 || i == 1) {\\n… See the full description on the dataset page: https://huggingface.co/datasets/Mxode/C-Language-Chat-Debug-Multiturn-Zh.Questions_Answers_In_Sinhala_Language@misc{AyeshaKalpani_2024,
title={Questions_Answers_In_Sinhala_Language},
author={Ayesha Kalpani},
year={2024},
url={},
}
Questions_Answers_In_Sinhala_Language
Dataset Description
A dataset containing questions and answers in the Sinhala language. This dataset is intended for training and evaluating question-answering models in Sinhala.
Dataset Details
License
This dataset is licensed under the MIT License.
Task… See the full description on the dataset page: https://huggingface.co/datasets/AyeshaKalpani98/Questions_Answers_In_Sinhala_Language.StackOverflow-QA-C-Language-5kPS: More data (40k) can be found here Mxode/StackOverflow-QA-C-Language-40k.
This is a collection of ~5000 QA's in C Language from StackOverflow. The data has been initially cleaned, and each response is with Accepted Answer.
All data is <500 in length.
The questions and answers were organized into a one-line format. A sample format is shown below:
{
"question": "```\nFILE* file = fopen(some file)\n\npcap_t* pd = pcap_fopen_offline(file)\n\npcap_close(pd)\n\nfclose(file)\n```\n\nThis… See the full description on the dataset page: https://huggingface.co/datasets/Mxode/StackOverflow-QA-C-Language-5k.mojo-programming-language-qnaA synthetic dataset from Claude Sonnet 3.5. The source documents are real pulled from the Mojo documentation, but everything else is synthetic.
low_resource_language
Beyond Log Likelihood Low Resource Language
This dataset bundle contains the low-resource language training/validation
parquet files and the MMLU-ProX-style multilingual multiple-choice test JSON
used by the Beyond-Log-Likelihood repository.
piqa_es
Dataset Card for PIQA (Spanish Version)
Dataset summary
This dataset provides the Spanish translation and adaptation of the validation
set of PIQA (Physical Interaction: Question Answering). The original dataset
was designed to evaluate physical commonsense reasoning in language models
through questions about everyday situations. Each example presents a physical
goal and two possible solutions, only one of which is correct.
This Spanish adaptation enables… See the full description on the dataset page: https://huggingface.co/datasets/langtech-languagemodeling/piqa_es.siqa_ca_old
Dataset Card: SIQA_CA (Pre-revision version)
Description
SIQA_CA (Pre-revision) is an earlier Catalan translation of the Social IQa (SIQA) dataset, a benchmark designed to evaluate commonsense reasoning about social interactions. This version consists of manually translated instances from the original English dataset into Catalan. It is used as a baseline for comparison against a revised and improved version of the dataset (SIQA_CA v2).
Motivation and Use Case… See the full description on the dataset page: https://huggingface.co/datasets/langtech-languagemodeling/siqa_ca_old.east-africa-language-dataset
East African Multilingual Dataset (EAAML)
Overview
EAAML is a multilingual instruction dataset focused on East African applications.
Domains:
Translation
Agriculture
Healthcare
Education
Daily conversation
Languages:
English
Swahili
Dataset Splits
train: 400 examples
validation: 50 examples
test: 50 examples
Fields
instruction
input
output
Example
{
"instruction": "Translate to Swahili",
"input": "Education… See the full description on the dataset page: https://huggingface.co/datasets/vancouverevs/east-africa-language-dataset.proxy-mt-benchmark-scores
Proxy-MT Benchmark Scores
Multilingual benchmark results for 50 open-weight LLMs, evaluated with the
lm-evaluation-harness via a vLLM
backend. Covers reasoning, comprehension, and knowledge tasks with an emphasis on
African and other lower-resource languages.
Layout
scores/<model>.csv # parsed per-language scores (tidy, ready to plot)
raw/<model>/.../results_*.json # raw lm-eval-harness result files
raw/<model>/raw_log.txt # full evaluation… See the full description on the dataset page: https://huggingface.co/datasets/African-Languages-Lab/proxy-mt-benchmark-scores.Indian_language_community_chatbot.csvlanguage