datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
mala-monolingual-integration
MaLA Corpus: Massive Language Adaptation Corpus
This is the noisy version that integrates texts from different sources.
Dataset Summary
The MaLA Corpus (Massive Language Adaptation) is a comprehensive, multilingual dataset designed to support the continual pre-training of large language models. It covers 939 languages and consists of over 74 billion tokens, making it one of the largest datasets of its kind. With a focus on improving the representation of… See the full description on the dataset page: https://huggingface.co/datasets/MaLA-LM/mala-monolingual-integration.mala-monolingual-split
MaLA Corpus: Massive Language Adaptation Corpus
This version contains train and validation splits.
Dataset Summary
The MaLA Corpus (Massive Language Adaptation) is a comprehensive, multilingual dataset designed to support the continual pre-training of large language models. It covers 939 languages and consists of over 74 billion tokens, making it one of the largest datasets of its kind. With a focus on improving the representation of low-resource languages, the… See the full description on the dataset page: https://huggingface.co/datasets/MaLA-LM/mala-monolingual-split.instructions
Merged Instructions Dataset
Merged Dataset for the response of instructions.
qg_squad[SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) evaluation set for the question generation (QG) models. The split
of test and development set follows the ["Neural Question Generation"](https://arxiv.org/abs/1705.00106) work and is
compatible with the [leader board](https://paperswithcode.com/sota/question-generation-on-squad11).qg_esquad[SQuAD-es](https://huggingface.co/datasets/squad_es) dataset for question generation (QG) task.qg_jaquad[JaQuAD](https://github.com/SkelterLabsInc/JaQuAD) dataset for question generation (QG) task. The test set of the original
data is not publicly released, so we randomly sampled test questions from the training set.qg_koquad[KorQuAD](https://huggingface.co/datasets/squad_kor_v1) dataset for question generation (QG) task.lmcache-agentic-traces
LMCache Agentic Dataset Collection
A curated dataset collection of 787 multi-turn agentic LLM sessions (24,881 total LLM iterations) designed for benchmarking stateful LLM serving systems. Every session exhibits at least 5 turns with prefix growth and builds to at least 10K tokens of context — making it ideal for evaluating tiered KV Cache solutions like LMCache.
Motivation
Modern LLM agents (coding assistants, research agents, tool-calling systems) make dozens of… See the full description on the dataset page: https://huggingface.co/datasets/sammshen/lmcache-agentic-traces.qg_subjqa[SubjQA](https://github.com/megagonlabs/SubjQA) dataset for question generation (QG) task.PolyWritePolyWrite is a novel multilingual dataset developed for evaluating open-ended generation across 240 languages. We use ChatGPT to create diverse prompts in English, and then use Google Translate to translate these prompts into various languages, enabling models to generate creative content in multilingual settings. The benchmark includes 31 writing tasks—such as storytelling and email writing—across 155 unique prompts. To ensure translation quality, we back-translate the multilingual prompts… See the full description on the dataset page: https://huggingface.co/datasets/MaLA-LM/PolyWrite.Wizard-LM-Chinese-instruct-evolWizard-LM-Chinese是在MSRA的Wizard-LM数据集上，对指令进行翻译，然后再调用GPT获得答案的数据集
Wizard-LM包含了很多难度超过Alpaca的指令。
中文的问题翻译会有少量指令注入导致翻译失败的情况
中文回答是根据中文问题再进行问询得到的。
我们会陆续将更多数据集发布到hf，包括
Coco Caption的中文翻译
CoQA的中文翻译
CNewSum的Embedding数据
增广的开放QA数据
WizardLM的中文翻译
如果你也在做这些数据集的筹备，欢迎来联系我们，避免重复花钱。
骆驼(Luotuo): 开源中文大语言模型
https://github.com/LC1332/Luotuo-Chinese-LLM
骆驼(Luotuo)项目是由冷子昂 @ 商汤科技, 陈启源 @ 华中师范大学 以及 李鲁鲁 @ 商汤科技 发起的中文大语言模型开源项目，包含了一系列语言模型。
( 注意: 陈启源 正在寻找2024推免导师，欢迎联系 )
骆驼项目不是商汤科技的官方产品。
Citation… See the full description on the dataset page: https://huggingface.co/datasets/silk-road/Wizard-LM-Chinese-instruct-evol.MathOlympiadBenchThis repository contains the MathOlympiadBench dataset, which is introduced in the paper Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction.
Project Page: https://blog.goedel-prover.com
Code Repository: https://github.com/Goedel-LM/Goedel-Prover-V2
MathOlympiadBench (Math Olympiad) comprises human-verified formalizations of Olympiad-level mathematical competition problems, sourced from Compfiles and IMOSLLean4 repository. MathOlympiadBench… See the full description on the dataset page: https://huggingface.co/datasets/Goedel-LM/MathOlympiadBench.mala-opus-dedup-2410-reLIDqg_squadshifts[SQuAD Shifts](https://modestyachts.github.io/squadshifts-website/index.html) dataset for question generation (QG) task.FineProofs-SFT
FineProofs SFT
Dataset Description
FineProofs SFT is a high-quality supervised fine-tuning dataset containing mathematical Olympiad problems paired with chain-of-thought reasoning and formal proofs distilled from DeepSeek-Math-V2. The dataset comprises 7,777 samples (4,300 unique problems) sourced from international Olympiad competitions and Art of Problem Solving (AoPS), each annotated with:
Detailed reasoning traces (thinking content) generated by… See the full description on the dataset page: https://huggingface.co/datasets/lm-provers/FineProofs-SFT.reasoning_gym_lmeh
Reasoning-gym tasks
This is an implementation of reasoning-gym into a fixed dataset to be used within lm-evaluation-harness ecosystem.
The dataset is meant to be used with semantic extraction (on most cases), applied by means of the a-vert method. Some higher level tasks (like 'codeio') use the native reasoning-gym methods to extract scores.
For each task 100 samples are geenrated and most instructions or hints are removed (we dont want to condition the LM answer).
Currently we… See the full description on the dataset page: https://huggingface.co/datasets/PNYX/reasoning_gym_lmeh.lmsys_chat_1m_clean_R1
oumi-ai/lmsys_chat_1m_clean_R1
lmsys_chat_1m_clean_R1 is a text dataset designed to train Conversational Language Models with DeepSeek-R1 level reasoning.
Prompts were pulled from LMSYS and filtered to lmsys_chat_1m_clean, and responses were taken from DeepSeek-R1 without additional filters present.
We release lmsys_chat_1m_clean_R1 to help enable the community to develop the best fully open reasoning model!
lmsys_chat_1m_clean queries with responses generated from… See the full description on the dataset page: https://huggingface.co/datasets/oumi-ai/lmsys_chat_1m_clean_R1.GlotStoryBook
Dataset Description
Story Books for 180 ISO-639-3 codes.
The Parallel ID or parallel_id can be used to find the parallel documents in different languages and build a parallel dataset.
This dataset consists of 2 subsets:
default, which consists of 4 publishers:
asp: African Storybook
pb: Pratham Books
lcb: Little Cree Books
lida: LIDA Stories
nalibali, which comes from Nal'ibali stories.
Usage (HF Loader)
default:
from datasets import load_dataset
dataset… See the full description on the dataset page: https://huggingface.co/datasets/cis-lmu/GlotStoryBook.lma_datasets
LMA Phase 1 --- Hindi and Nepali pretraining corpora
Two monolingual corpora built for a pair of ~25M-parameter decoder-only
Transformers. Hindi is the higher-resource language, Nepali the lower-resource
one. Both are written in Devanagari (U+0900-U+097F), so script cannot be used
to tell them apart --- separating them is the central technical problem this
dataset solves rather than assumes.
language
documents
characters
manual (chars)
tokens
manual (tokens)
train
val
test… See the full description on the dataset page: https://huggingface.co/datasets/meet5568/lma_datasets.chinese-small-lm-corpus
Chinese Small LM Corpus
用于中文小型语言模型预训练的统一文本语料，字段为：
text：规范化后的训练文本
source：原始数据集名称
数据量
来源
有效样本数
TinyStories-Zh-2M
1,994,291
Wikipedia-20231101.zh
1,384,748
Zhihu-KOL
1,002,863
总计
4,381,902
来源与许可
RobinChen2001/TinyStories-Zh-2M：数据卡标注 MIT；同时应检查英文上游数据及机器翻译来源条款。
wikimedia/wikipedia (20231101.zh)：CC BY-SA 3.0 与 GFDL。
wangrui6/Zhihu-KOL：原数据卡未声明许可证。
此合并数据集不提供统一的再授权。下载者须分别遵守各来源的许可、署名、隐私与内容使用要求。
LLaVA-OneVision-Mid-Data
Dataset Card for LLaVA-OneVision
Due to unknow reasons, we are unable to process dataset with large amount into required HF format. So we directly upload the json files and image folders (compressed into tar.gz files).
You can use the following link to directly download and decompress them.
https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Mid-Data/tree/main/evol_instruct
We provide the whole details of LLaVA-OneVision Dataset. In this dataset, we include the data splits… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Mid-Data.task137_detoxifying-lms_classification_toxicity
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task137_detoxifying-lms_classification_toxicity
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task137_detoxifying-lms_classification_toxicity.qag_tweetqaQuestion & answer generation dataset based on [TweetQA](https://huggingface.co/datasets/tweet_qa).update-interrupt-benchmark
Update-Driven Math & Code Interrupt Datasets
Paper: Are Large Reasoning Models Interruptible?
Authors: Tsung-Han Wu*, Mihran Miroyan*, David Chan, Trevor Darrell, Narges Norouzi, Joseph Gonzalez
Project page: https://dynamic-lm.github.io/
Github: https://github.com/dynamic-lm/interrupt-lrm
This dataset page contains the update-driven interrupt subsets for math (GSM8K, MATH500, AIME) and coding (LiveCodeBench) problems. For both splits, we revise the source problems and… See the full description on the dataset page: https://huggingface.co/datasets/dynamic-lm/update-interrupt-benchmark.SFT_Dataset
Pythagoras SFT Dataset
Project Page | GitHub | Paper
Data
Our training dataset consists of approximately 841K problems paired with Lean formal statements, formal proofs, and reasoning chains. We release a partial subset, which consists of 126K instances:
30K easy instances
49K medium instances
47K hard instances
Complete data will be released soon.
The complete explanation of the synthetic data generation pipeline can be found in Pythagoras-Prover: Advancing… See the full description on the dataset page: https://huggingface.co/datasets/Pythagoras-LM/SFT_Dataset.simple_wikipedia_LM
Dataset Card for "simple_wikipedia_LM"
A filtered/edited version of pszemraj/simple_wikipedia that removes headings/contents that appear in the text column without any relevant text for them (at least in the simple split).
import re
def split_on_headings(text):
headings = ["References", "Related pages", "Other websites", "Further reading"]
for heading in headings:
parts = re.split(
r"^\s*" + re.escape(heading) + r".*$", text, flags=re.MULTILINE… See the full description on the dataset page: https://huggingface.co/datasets/pszemraj/simple_wikipedia_LM.qg_tweetqaQuestion generation dataset based on [TweetQA](https://huggingface.co/datasets/tweet_qa).task140_detoxifying-lms_classification_style
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task140_detoxifying-lms_classification_style
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task140_detoxifying-lms_classification_style.lm1b
LM1B - One Billion Word Benchmark
Dataset Description
The One Billion Word Benchmark is a large language modeling dataset.
It contains approximately one billion words of training data derived from news articles.
How was this dataset built?
We download the full LM1B dataset from TensorFlow Datasets (TFDS) and convert it to HuggingFace format automatically. The full script is in lm1b.py. The required environment is:
tensorflow==2.20.0
tensorflow-datasets==4.9.9… See the full description on the dataset page: https://huggingface.co/datasets/FrankCCCCC/lm1b.lmcache-agentic-traces
LMCache Agentic Dataset Collection
A curated dataset collection of 787 multi-turn agentic LLM sessions (24,881 total LLM iterations) designed for benchmarking stateful LLM serving systems. Every session exhibits at least 5 turns with prefix growth and builds to at least 10K tokens of context — making it ideal for evaluating tiered KV Cache solutions like LMCache.
Motivation
Modern LLM agents (coding assistants, research agents, tool-calling systems) make dozens of… See the full description on the dataset page: https://huggingface.co/datasets/DiscoPosse/lmcache-agentic-traces.