datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
smollm-corpus
SmolLM-Corpus
This dataset is a curated collection of high-quality educational and synthetic data designed for training small language models.
You can find more details about the models trained on this dataset in our SmolLM blog post.
Dataset subsets
Cosmopedia v2
Cosmopedia v2 is an enhanced version of Cosmopedia, the largest synthetic dataset for pre-training, consisting of over 39 million textbooks, blog posts, and stories generated by… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus.smoltalk
SmolTalk
Dataset description
This is a synthetic dataset designed for supervised finetuning (SFT) of LLMs. It was used to build SmolLM2-Instruct family of models and contains 1M samples. More details in our paper https://arxiv.org/abs/2502.02737
During the development of SmolLM2, we observed that models finetuned on public SFT datasets underperformed compared to other models with proprietary instruction datasets. To address this gap, we created new synthetic datasets… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/smoltalk.finemath
📐 FineMath
What is it?
📐 FineMath consists of 34B tokens (FineMath-3+) and 54B tokens (FineMath-3+ with InfiMM-WebMath-3+) of mathematical educational content filtered from CommonCrawl. To curate this dataset, we trained a mathematical content classifier using annotations generated by LLama-3.1-70B-Instruct. We used the classifier to retain only the most educational mathematics content, focusing on clear explanations and step-by-step problem solving rather than… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/finemath.cosmopedia
Cosmopedia v0.1
Image generated by DALL-E, the prompt was generated by Mixtral-8x7B-Instruct-v0.1
Note: Cosmopedia v0.2 is available at smollm-corpus
User: What do you think "Cosmopedia" could mean? Hint: in our case it's not related to cosmology.
Mixtral-8x7B-Instruct-v0.1: A possible meaning for "Cosmopedia" could be an encyclopedia or collection of information about
different cultures, societies, and topics from around the world, emphasizing diversity and global… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/cosmopedia.smoltalk2
SmolTalk2
Dataset description
This dataset contains three subsets (Mid, SFT, Preference) that correspond to the three phases of Post-Training for SmolLM3-3B. You can find more details in our blog post about how we used the data in each of the stages SmolLM3.
The specific weight of each subset is available in the training recipe in SmolLM's repository.
You can load a dataset using
from datasets import load_dataset
# To load the train split of a specific subset, such as… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/smoltalk2.imagessmol-smoltalk
Smol-SmalTalk
This is a subset of SmolTalk dataset adapted for smol models with less than 1B parameters. We used it to build SmolLM2-360M-Instruct and
SmolLM2-135M-Instruct. We do SFT on this dataset and then DPO on UltraFeedback.
Compared to SmolTalk:
The conversations from Smol-Magpie-Ultra are shorter in this dataset
We include less task specific data compared to SmolTalk (e.g no function calling and less rewriting and summarization examples) since these smaller models have… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk.stack-edu
💻 Stack-Edu
Stack-Edu is a 125B token dataset of educational code filtered from The Stack v2, precisely the curated training corpus of StarCoder2 models denoted StarCoder2Data. It is intended for Language Models training.
This dataset was curated using a classifier-based filtering strategy, inspired by 📚 FineWeb-Edu, to retain only the highest-quality educational programming content.
Stack-Edu shows consistent improvement over StarCoder2data on all the programming languages on… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/stack-edu.stackexchange_2025_mdopenstax_paragraphsTexbooks from openstax.org with their chapters, abstracts and sections.
Sample:
{
"book_title":"World History Volume 1, to 1500",
"language":"en",
"chapters":[
{
"title":"Preface",
"abstract":"None",
"sections":[
{
"title":"About OpenStax",
"paragraph":"OpenStax is part of Rice University, which is a 501(c)(3) nonprofit..."
},
{
"title":"About OpenStax Resources"… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/openstax_paragraphs.dclm-edu
DCLM-Edu
Description
This is a filtered version of DCLM dataset using FineWeb-Edu educational quality classifier. We annotate each web page based on the educational quality
on a scale from 0 to 5 and only keep samples with a score higher than 2. This dataset is intended for small language models training and was used to train SmolLM2-135M and SmolLM2-360M.
Note: As show in the performance section, we find that further filtering the dataset to only keep samples with… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/dclm-edu.everyday-conversations-llama3.1-2k
Everyday conversations for Smol LLMs finetunings
This dataset contains 2.2k multi-turn conversations generated by Llama-3.1-70B-Instruct. We ask the LLM to generate a simple multi-turn conversation, with 3-4 short exchanges, between a User and an AI Assistant about a certain topic.
The topics are chosen to be simple to understand by smol LLMs and cover everyday topics + elementary science. We include:
20 everyday topics with 100 subtopics each
43 elementary science topics with 10… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k.cosmopedia-100k
Dataset description
This is a 100k subset of Cosmopedia dataset. A synthetic dataset of textbooks, blogposts, stories, posts and WikiHow articles generated by Mixtral-8x7B-Instruct-v0.1.
Here's how you can load the dataset
from datasets import load_dataset
ds = load_dataset("HuggingFaceTB/cosmopedia-100k", split="train")
MATHMATH is a dataset of 12,500 challenging competition mathematics problems. Each
problem in Math has a full step-by-step solution which can be used to teach
models to generate answer derivations and explanations.OpenR1-Math-220k-default-verifiedissues-kaggle-notebooks
GitHub Issues & Kaggle Notebooks
Description
GitHub Issues & Kaggle Notebooks is a collection of two code datasets intended for language models training, they are sourced from GitHub issues and notebooks in Kaggle platform. These datasets are a modified part of the StarCoder2 model training corpus, precisely the bigcode/StarCoder2-Extras dataset. We reformat the samples to remove StarCoder2's special tokens and use natural text to delimit comments in issues and display… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/issues-kaggle-notebooks.python-edu-annotations
Annotations for 📚 Python-Edu classifier
This dataset contains the annotations used for training Python-Edu educational quality classifier. We prompt Llama-3-70B-Instruct to score python programs from StarCoderData based on their educational value.
Note: the dataset contains the Python program, the prompt (using the first 1000 characters of the program) and the scores but it doesn't contain the full Llama 3 generation.
smollm3-configs
SmolLM3 Training Configs
[IMPORTANT NOTE]: for the latest configs go to this repo: https://github.com/huggingface/smollm/tree/main/text/pretraining/smollm3
Here you can find the training configs for SmoLLM3-3B-Base using nanotron with exact training details and data mixtures.
The model was trained on 11.2T tokens in 3 stages on 4k context:
stage 1 config
stage 2 config
stage 3 config
And then we trained on an additional 2 stages to extend the contetx length to 64k:
stage 4… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs.smoltalk-multilingual8-Qwen3-32B-main-genMagPie-Pro-300k-MTcosmopedia_web_textbooksdetails_HuggingFaceTB__SmolLM2-1.7B-Instruct
Dataset Card for Evaluation run of HuggingFaceTB/SmolLM2-1.7B-Instruct
Dataset automatically created during the evaluation run of model HuggingFaceTB/SmolLM2-1.7B-Instruct.
The dataset is composed of 7 configuration, each one corresponding to one of the evaluated task.
The dataset has been created from 12 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest… See the full description on the dataset page: https://huggingface.co/datasets/SaylorTwift/details_HuggingFaceTB__SmolLM2-1.7B-Instruct.instruct-data-basics-smollm-H4Datasets of basic instructions and answers for SmolLM-Instruct models trainings: it includes answers to greetings and questions such as "Who are you". This dataset was included in training of SmolLM-Instruct v0.2 but we didn't notice that it had an impact on model generations.
We recommend using this generic larger dataset of multi-turn everyday conversations: https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k
OpenHermes-2.5-H4OpenHermes2.5 dataset formatted to be compatible with the alignement-handbook for SFT.
smollm3-blueprintHere you can find the SmolLM3 Engineering Blueprint
HuggingFaceTB-cosmopedia-cntraining-guide-nanotron-configsThis repository contains the nanotron training configs for the ablations in The Smol Training Guide.
self-oss-instruct-sc2-H4StarCoder2-Self-Instruct-OSS-50k dataset formatted to be compatible with the alignement-handbook for SFT.
ultrachat_questions_about_world
Ultrachat, Questions about the world
This is the "Questions about the world" subset of UltraChat, found in the this GitHub repo.
bisac_expanded_final