datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
stack-v3-train
🥞 The Stack v3
What is it?
What is being released
How to download and use it
Dataset statistics
Dataset structure
Dataset creation
Considerations for using the data
Additional information
What is it?
The Stack v3 is the largest, most up-to-date open dataset of source code, crawled directly from GitHub and built to pre-train code LLMs with full-repository context. It is the successor to The Stack v2 and, like its predecessor, is released to make the training… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train.the-stack-smol
Dataset Description
A small subset (~0.1%) of the-stack dataset, each programming language has 10,000 random samples from the original dataset. The dataset has 2.6GB of text (code).
Languages
The dataset contains 30 programming languages:
"assembly", "batchfile", "c++", "c", "c-sharp", "cmake", "css", "dockerfile", "fortran", "go", "haskell", "html", "java",
"javascript", "julia", "lua", "makefile", "markdown", "perl", "php", "powershell", "python", "ruby", "rust"… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/the-stack-smol.the-stack-dedup
Dataset Card for The Stack
Changelog
Release
Description
v1.0
Initial release of the Stack. Included 30 programming languages and 18 permissive licenses. Note: Three included licenses (MPL/EPL/LGPL) are considered weak copyleft licenses. The resulting near-deduplicated dataset is 1.5TB in size.
v1.1
The three copyleft licenses ((MPL/EPL/LGPL) were excluded and the list of permissive licenses extended to 193 licenses in total. The list of programming… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/the-stack-dedup.Stack-scBaseCount189Mstackoverflow-posts
StackOverflow Posts Markdown
Dataset Summary
This dataset contains all posts submitted to StackOverflow before the 14th of June 2023 formatted as Markdown text.
The dataset contains ~60 Million posts, totaling ~35GB in size and ~65 billion characters of text.
The data is sourced from Internet Archive StackExchange Data Dump.
Dataset Structure
Each record corresponds to one post of a particular type.
Original ordering from the data dump is not exactly preserved… See the full description on the dataset page: https://huggingface.co/datasets/mikex86/stackoverflow-posts.the-stack-v2-smollm3
The Stack v2 — materialized source code
Upstream dataset:
bigcode/the-stack-v2
Exact upstream commit:
e565caa3a78c2423bd374333a472b049eb090e47
Primary source-content endpoint:
https://softwareheritage.s3.amazonaws.com/content/{blob_id}
Configurations
TypeScript
Swift
Ruby
Rust
Go
Shell
Jupyter_Notebook
HTML
Python
Java
JavaScript
C
C++
C-Sharp
PHP
SQL
Markdown
Added columns
content: decoded source content
download_error: null on successful… See the full description on the dataset page: https://huggingface.co/datasets/jordangong/the-stack-v2-smollm3.the-stack
Dataset Card for The Stack
Changelog
Release
Description
v1.0
Initial release of the Stack. Included 30 programming languages and 18 permissive licenses. Note: Three included licenses (MPL/EPL/LGPL) are considered weak copyleft licenses. The resulting near-deduplicated dataset is 3TB in size.
v1.1
The three copyleft licenses ((MPL/EPL/LGPL) were excluded and the list of permissive licenses extended to 193 licenses in total. The list of programming… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/the-stack.stack-exchange-preferences
Dataset Card for H4 Stack Exchange Preferences Dataset
Dataset Summary
This dataset contains questions and answers from the Stack Overflow Data Dump for the purpose of preference model training.
Importantly, the questions have been filtered to fit the following criteria for preference models (following closely from Askell et al. 2021): have >=2 answers.
This data could also be used for instruction fine-tuning and language model training.
The questions are grouped with… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences.stackv2
Stack V2
Description
We filter the Stack V2 to only include code from openly licensed repositories, based on the license detection performed by the creators of Stack V2.
When multiple licenses are detected in a single repository, we ensure that all of the licenses are on the Blue Oak Council certified license list.
Per-document license information is available in the license entry of the metadata field of each example.
Code for collecting, processing, and preparing this… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/stackv2.Stack-RepoThis is the Stack-Repo datasetthe-stack-v2
The Stack v2
The dataset consists of 4 versions:
bigcode/the-stack-v2: the full "The Stack v2" dataset <-- you are here
bigcode/the-stack-v2-dedup: based on the bigcode/the-stack-v2 but further near-deduplicated
bigcode/the-stack-v2-train-full-ids: based on the bigcode/the-stack-v2-dedup dataset but further filtered with heuristics and spanning 600+ programming languages. The data is grouped into repositories.
bigcode/the-stack-v2-train-smol-ids: based on the… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/the-stack-v2.stack-edu
💻 Stack-Edu
Stack-Edu is a 125B token dataset of educational code filtered from The Stack v2, precisely the curated training corpus of StarCoder2 models denoted StarCoder2Data. It is intended for Language Models training.
This dataset was curated using a classifier-based filtering strategy, inspired by 📚 FineWeb-Edu, to retain only the highest-quality educational programming content.
Stack-Edu shows consistent improvement over StarCoder2data on all the programming languages on… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/stack-edu.the-stack-smol-xl
Dataset Description
A small subset of the-stack dataset, with 87 programming languages, each has 10,000 random samples from the original dataset.
Languages
The dataset contains 87 programming languages:
'ada', 'agda', 'alloy', 'antlr', 'applescript', 'assembly', 'augeas', 'awk', 'batchfile', 'bison', 'bluespec', 'c',
'c++', 'c-sharp', 'clojure', 'cmake', 'coffeescript', 'common-lisp', 'css', 'cuda', 'dart', 'dockerfile', 'elixir',
'elm', 'emacs-lisp','erlang'… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/the-stack-smol-xl.the-stack-v2-prstackv2_edu_filtered
Stack V2 Edu
Description
We filter the Stack V2 to only include code from openly licensed repositories, based on the license detection performed by the creators of Stack V2. When multiple licenses are detected in a single repository, we ensure that all of the licenses are on the Blue Oak Council certified license list. Per-document license information is available in the license entry of the metadata field of each example. Code for collecting, processing, and preparing… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/stackv2_edu_filtered.stackexchange-clustering
StackExchangeClustering.v2
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of titles from 121 stackexchanges. Clustering of 25 sets, each with 10-50 classes, and each class with 100 - 1000 sentences.
Task category
t2c
Domains
Web, Written
Reference
https://arxiv.org/abs/2104.07081
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/stackexchange-clustering.StackExchange_Mar2023
Dataset Card for "StackExchange_Mar2023"
More Information needed
the-stack-smol-xs\the-stack-metadata
Dataset Card for The Stack Metadata
Changelog
Release
Description
v1.1
This is the first release of the metadata. It is for The Stack v1.1
v1.2
Metadata dataset matching The Stack v1.2
Dataset Summary
This is a set of additional information for repositories used for The Stack. It contains file paths, detected licenes as well as some other information for the repositories.
Supported Tasks and Leaderboards
The main… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/the-stack-metadata.stackoverflow-with-meta-data
Dataset Card for "stackoverflow-with-meta-data"
More Information needed
stackexchange
Stack Exchange
Description
StackExchange is a collection of Q&A communities spanning a wide variety of topics.
While StackExchange formerly provided structured dumps of all of their content, since July of 2024, StackExchange has stopped publishing XML dumps to the Internet Archive.
Instead, each site can provide a logged in user with a custom url to download the dump for that site.
This means that dumps for defunct sites like windowsphone.stackexchange.com are… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/stackexchange.StackV1-popular
Stack V1 with popular programming languages
Javascript
Python
C
C++
SQL
Cuda
stackexchange_2025_mdstack-exchange-paired
StackExchange Paired
This is a processed version of the HuggingFaceH4/stack-exchange-preferences. The following steps were applied:
Parse HTML to Markdown with markdownify
Create pairs (response_j, response_k) where j was rated better than k
Sample at most 10 pairs per question
Shuffle the dataset globally
This dataset is designed to be used for preference learning. The processing notebook is in the repository as well.
dataset-the-stack-v2-dedup-sub
The Stack v2 Subset with File Contents (Python, Java, JavaScript, C, C++)
TempestTeam/dataset-the-stack-v2-dedup-sub
Dataset Summary
This dataset is a language-filtered and self-contained subset of bigcode/the-stack-v2-dedup, part
of the BigCode Project.
It contains only files written in the following programming languages:
Python 🐍
Java ☕
JavaScript 📜
C ⚙️
C++ ⚙️
Unlike the original dataset, which only includes metadata and Software Heritage IDs, this subset includes… See the full description on the dataset page: https://huggingface.co/datasets/Cyrile/dataset-the-stack-v2-dedup-sub.Stack-CellxGene45M
CellxGene 45M Collection
A curated subset of CellxGene (~45M cells) used to align the Stack model after pretraining on full human scBaseCount.
Selection Criteria
≥ 50,000 cells per dataset
≥ 5 donors per dataset
Cell Type Annotations
Author-annotated coarse-grained cell type labels were heuristically identified and transferred to adata.obs["author_cell_type"].
stackexchange_filtered
Stack Exchange
Description
StackExchange is a collection of Q&A communities spanning a wide variety of topics.While StackExchange formerly provided structured dumps of all of their content, since July of 2024, StackExchange has stopped publishing XML dumps to the Internet Archive.Instead, each site can provide a logged-in user with a custom URL to download the dump for that site.This means that dumps for defunct sites like windowsphone.stackexchange.com are… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/stackexchange_filtered.Stack_Tokenizedstack-exchange-instruction
Dataset Card for "stack-exchange-instruction"
More Information needed
stackexchange-clustering-p2p
StackExchangeClusteringP2P.v2
An MTEB dataset
Massive Text Embedding Benchmark
Clustering of title+body from stackexchange. Clustering of 5 sets of 10k paragraphs and 5 sets of 5k paragraphs.
Task category
t2c
Domains
Web, Written
Reference
https://arxiv.org/abs/2104.07081
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["StackExchangeClusteringP2P.v2"])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/stackexchange-clustering-p2p.