HuggingFaceCode/stack-v3-train

1---2thumbnail: "https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train/resolve/main/assets/banner.png"3annotations_creators: []4language_creators:5- crowdsourced6- expert-generated7language:8- code9license:10- odc-by11multilinguality:12- multilingual13size_categories:14- 100M<n<1B15source_datasets: []16task_categories:17- text-generation18task_ids: []19pretty_name: The Stack v320tags:21- code22dataset_info:23  features:24  - name: repo_path25    dtype: string26  - name: repo_id27    dtype: int6428  - name: commit_id29    dtype: string30  - name: github_metadata31    struct:32    - name: branch33      dtype: string34    - name: commit_count35      dtype: int3236    - name: repo_created_at37      dtype: string38    - name: is_fork39      dtype: bool40    - name: is_org_owned41      dtype: bool42    - name: forked_from43      dtype: string44    - name: stars45      dtype: int3246    - name: forks47      dtype: int3248    - name: issues49      dtype: int3250    - name: pull_requests51      dtype: int3252  - name: num_files53    dtype: int6454  - name: files55    list:56    - name: content_id57      dtype: string58    - name: content59      dtype: string60    - name: size_bytes61      dtype: int3262    - name: file_path63      dtype: string64    - name: file_timestamp65      dtype: int6466    - name: language67      dtype: string68    - name: is_vendor69      dtype: bool70    - name: license_type71      dtype: string72    - name: detected_licenses73      sequence: string74configs:75- config_name: default76  data_files:77  - split: train78    path: data/*.parquet79---80 81<img src="./assets/banner.png" width="100%" alt="The Stack v3 banner">82 83# 🥞 The Stack v384 85- [What is it?](#what-is-it)86- [What is being released](#what-is-being-released)87- [How to download and use it](#how-to-download-and-use-it)88- [Dataset statistics](#dataset-statistics)89- [Dataset structure](#dataset-structure)90- [Dataset creation](#dataset-creation)91- [Considerations for using the data](#considerations-for-using-the-data)92- [Additional information](#additional-information)93 94## What is it?95 96The Stack v3 is the **largest, most up-to-date open dataset of source code**, crawled directly from GitHub and built to pre-train code LLMs with full-repository context. It is the successor to [The Stack v2](https://huggingface.co/datasets/bigcode/the-stack-v2) and, like its predecessor, is released to make the training of code models more **open, reproducible, and transparent**.97 98If your code is included in this version and you would like it removed for any reason, please use the ["Am I in The Stack?"](https://huggingface.co/spaces/HuggingFaceCode/in-the-stack) Space to submit a removal request. Repositories that opted out are removed from the dataset before each patch release.99 100Two things set v3 apart from earlier releases of The Stack:101 1021. **The file contents are included inline.** The decoded UTF-8 source text is embedded directly in the dataset, so it is fully self-contained — you can start training the moment the download finishes.1032. **It reflects the state of GitHub in August 2025.** The corpus is a direct crawl of GitHub repositories at their default-branch HEAD, capturing roughly two additional years of open-source code compared to The Stack v2.104 105> The Stack v3 training subset contains **15.9 TB** of source code across **713** programming languages from **173M** repositories (~**3.6 trillion** tokens).106> The Full Stack v3 corpus contains **113.7 TB** of source code across **770** languages from **224M** repositories.107 108### The Stack across versions109 110| | The Stack v1 | The Stack v2 | The Stack v3 |111|-|-|-|--------------|112| cutoff | 2022 | 2023 | 2025         |113| full | 6.4 TB | 67.5 TB | 113.7 TB     |114| dedup | 2.9 TB | 32.1 TB | 99.3 TB      |115| train | ~200B tokens | 2.0 TB / ~550B tokens | 11.5 TB / ~3.6T tokens |116| languages | 358 | 618 | 713          |117 118## What is being released119 120The Stack v3 is published in two complementary datasets:121 122| Dataset                                                                                      | What it is | Best for |123|----------------------------------------------------------------------------------------------|---|---|124| [`HuggingFaceCode/stack-v3-train`](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train) | Near-deduplicated, heuristically filtered code, **grouped by repository**, with file contents inline. Ready for training. | Pre-training code LLMs with repository context. |125| [`HuggingFaceCode/stack-v3-full`](https://huggingface.co/buckets/HuggingFaceCode/stack-v3-full)          | The complete corpus before quality filtering and near-dedup selection. Hosted as an HF Storage Bucket. Split into a **`metadata`** part (repo/file metadata) and a **`contents`** part (one row per unique file blob). | Research on the raw corpus, custom filtering, or building your own training mix. |126 127`stack-v3-train` is derived from `stack-v3-full`: it is the near-deduplicated corpus with an additional round of quality heuristics applied, restricted to one representative per duplicate cluster, and grouped so that each row is a full repository.128 129`stack-v3-full` is documented in its own [dataset card](https://huggingface.co/buckets/HuggingFaceCode/stack-v3-full). Reach for it when `stack-v3-train` is not enough: it retains all near-duplicates with their **`dedup_cluster`** IDs and pre-filtering signals (so you can roll your own deduplication and filtering), and its metadata lists **every** file of every repository — including stubs for binary, oversized, and undetected-language files — so you can locate and re-crawl anything we excluded (PDFs, binary assets, niche languages).130 131### Changelog132 133| Release | Description |134|---|---|135| v3.1 | Removed exact-duplicate files that leaked due to a partitioning bug, 4.9T -> 3.6T tokens. |136| v3.0 | Initial release of The Stack v3. |137 138## How to download and use it139 140### `stack-v3-train`141 142Each row is a **repository**; the code lives inline under `files[].content`.143 144```python145from datasets import load_dataset146 147# Full dataset (downloads locally)148ds = load_dataset("HuggingFaceCode/stack-v3-train", split="train", cache_dir="/path/to/local/cache", num_proc=16)149 150# Streaming (downloads shards on demand — recommended for quick tests)151ds = load_dataset("HuggingFaceCode/stack-v3-train", split="train", streaming=True)152 153for repo in ds:154    print(repo["repo_path"], "-", repo["num_files"], "files")155    for f in repo["files"]:156        print(f["file_path"], f["language"], f["license_type"])157        print(f["content"][:100], "...") 158    break159```160 161> [!TIP]162> There are no per-language configs: each row is a whole repository, and its files (usually a mix of languages) live in the `files[]` array. To filter by language, iterate the dataset (streaming works well) and keep the files you want, e.g. `[f for f in repo["files"] if f["language"] == "Python"]`. If you need language-partitioned files, use [`stack-v3-full`](https://huggingface.co/buckets/HuggingFaceCode/stack-v3-full), where contents are partitioned by language.163 164For analytical access without loading through the `datasets` library, you can query the Parquet files directly with DuckDB or Polars:165 166```python167import duckdb168from huggingface_hub import HfFileSystem169 170duckdb.register_filesystem(HfFileSystem())171 172duckdb.sql("""173    SELECT repo_path, num_files174    FROM 'hf://datasets/HuggingFaceCode/stack-v3-train/data/*.parquet'175    LIMIT 10176""").show()177```178 179### `stack-v3-full`180 181`stack-v3-full` is hosted as a [Hugging Face Storage Bucket](https://huggingface.co/docs/hub/storage-buckets) (not a regular dataset repo), split into a `metadata/` part (one row per repository) and a `contents/` part (one row per unique file blob, partitioned by language) that join on `content_id`. See [its dataset card](https://huggingface.co/buckets/HuggingFaceCode/stack-v3-full) for access instructions, join examples (DuckDB, pandas, PySpark), the full schema, and how stubs and `dedup_cluster` work.182 183## Dataset statistics184 185<img src="./assets/stack_v3_multiplier_light.png" alt="The Stack v3 compared to previous versions" style="max-width: min(100%, 720px);">186 187### Languages188 189The dataset contains **713** programming and markup languages.190 191Per-language breakdowns are available as JSON:192 193* **Train split:** [`stats/train/stats_by_language.json`](./stats/train/stats_by_language.json) — 713 languages194* **Full (unfiltered):** [`stats/full/stats_by_language.json`](./stats/full/stats_by_language.json) — 770 languages195 196## Dataset structure197 198### `stack-v3-train` — data fields199 200Each row is a **repository**.201 202* `repo_path` (`string`): `owner/name` slug of the repository on GitHub.203* `repo_id` (`int64`): Numeric GitHub repository ID.204* `commit_id` (`string`): Commit (HEAD of the default branch) the snapshot was taken from.205* `github_metadata` (`struct`): Repository-level metadata from the GitHub repository page:206  * `branch` (`string`): Default branch name.207  * `commit_count` (`int32`): Number of commits on the default branch.208  * `repo_created_at` (`string`): Repository creation timestamp.209  * `is_fork` (`bool`): Whether the repository is a fork.210  * `is_org_owned` (`bool`): Whether the repository is owned by an organization.211  * `forked_from` (`string`): Source repository if this is a fork, else `None`.212  * `stars` (`int32`): Star count at crawl time.213  * `forks` (`int32`): Fork count at crawl time.214  * `issues` (`int32`): Open issue count at crawl time.215  * `pull_requests` (`int32`): Open pull-request count at crawl time.216* `num_files` (`int64`): Number of files in this repository row.217* `files` (`list[struct]`): The files of the repository:218  * `content_id` (`string`): SHA-1 of the file content.219  * `content` (`string`): The UTF-8 file content (PII-redacted).220  * `size_bytes` (`int32`): Length of the content in UTF-8 bytes.221  * `file_path` (`string`): Path of the file within the repository.222  * `file_timestamp` (`int64`): File modification time (Unix seconds). Only as accurate as the GitHub metadata.223  * `language` (`string`): Programming language, detected by [`go-enry`](https://github.com/go-enry/go-enry) (a port of GitHub's Linguist).224  * `is_vendor` (`bool`): Whether the file is a vendored/third-party file (per `go-enry`).225  * `license_type` (`string`): `permissive` or `no_license`.226  * `detected_licenses` (`list[string]`): SPDX license identifiers detected by [ScanCode](https://github.com/nexB/scancode-toolkit).227 228### `stack-v3-full` — data fields229 230`stack-v3-full` shares the same repository-level fields, but its `files[]` structs reference content by `content_id` instead of embedding it, and include **stub** entries for excluded files (binary, oversized, undetected language). The full schema and stub semantics are documented in the [`stack-v3-full` dataset card](https://huggingface.co/buckets/HuggingFaceCode/stack-v3-full#data-fields).231 232## Dataset creation233 234The Stack v3 was built in a similar fashion as The Stack v2 — language detection with [`go-enry`](https://github.com/go-enry/go-enry), file-level license detection with [ScanCode](https://github.com/nexB/scancode-toolkit), MinHash-LSH near-deduplication, PII redaction, quality filtering, and repository-level grouping — with updated tool versions and a few meaningful changes:235 236### What changed from v2237 238| | The Stack v2 | The Stack v3 |239|---|---|---|240| **Source** | Software Heritage graph (2023) | Direct GitHub crawl (cutoff: 2025-08-07) |241| **Content** | File IDs only | File contents inline — self-contained |242| **Near-dedup** | Per-language | Language-agnostic, with Jaccard verification after connected-component clustering (no false positives) |243 244The rest of the pipeline (license detection, quality heuristics, PII redaction, grouping) follows the same methodology as v2 with updated tool versions.245 246> **Scope note:** This release focuses on **source code and Jupyter notebooks**. It does not include the additional non-code sources that accompanied StarCoder2 training (GitHub issues, pull requests, Kaggle notebooks, documentation, ArXiv, Wikipedia, OpenWebMath, etc.).247 248### Curation rationale249 250Most code LLMs are trained on data that is never released, making the systems hard to reproduce and study. By publishing a large, contemporary, contents-included code dataset together with its construction code, we aim to make code-LLM pre-training reproducible and to give researchers a transparent, up-to-date corpus to build on.251 252### Source data253 254#### Data collection255 256The list of repositories to crawl was assembled by taking the union of:257 258- Every GitHub repository that appears in [GH Archive](https://www.gharchive.org/) public event data, and259- Every GitHub origin recorded in the Software Heritage graph (origin table, `2025-05-18` snapshot).260 261For each repository, the crawler fetched the GitHub repository page (for metadata such as stars, forks, and the HEAD commit) and downloaded a **single snapshot tarball** of the default branch at its HEAD commit — no git history is collected. Forks were only downloaded when they had at least **5 stars**.262 263During download, the following were dropped or emptied:264 265* Files larger than **5 MB**.266* Binary files (files containing a null byte).267* Symlinks and non-regular files.268* Repositories were capped at **1,000,000 files**.269 270 271The crawl completed on **August 7, 2025**. The dataset reflects the state of public GitHub repositories as of that date.272 273The crawl collected snapshots from **224M** repositories containing **43.9 billion** files (including stubs for binary, oversized, and undetected-language files).274 275The exact snapshot for every repository is traceable via its `commit_id`.276 277#### Cleaning and filtering278 279The corpus then passes through the following stages (implemented in Spark):280 2811. **Decoding & language detection.** File bytes are decoded to UTF-8 (falling back to `cchardet` encoding detection). Language, vendor, and generated-file flags are assigned with [`go-enry`](https://github.com/go-enry/go-enry). Each unique content blob is content-addressed by its SHA-1 (`content_id`) and stored once.2822. **License detection.** We detect licenses at the file level using the [ScanCode Toolkit](https://github.com/nexB/scancode-toolkit) and propagate them through the repository file tree:283   1. *Identify license files.* Any file whose name matches a license-bearing pattern (e.g. `LICENSE`, `COPYING`, `MIT.txt`, `Apache2.0`, `NOTICE`, `README`, `GUIDELINES`, …) is selected for scanning.284   2. *Detect licenses.* ScanCode's license detection runs on the content of each matching file and returns the SPDX identifiers of any licenses found.285   3. *Propagate to siblings.* The detected licenses are propagated to every file that shares the same directory prefix as the license file — e.g. a `LICENSE` at the repo root covers all files, while one in `src/lib/` covers only files under that subtree. When a file is covered by multiple license files from different directories, their licenses are merged.286   4. *Classify.* Each file is labelled `permissive` (at least one permissive license detected, no conflicting non-permissive license), `no_license` (no licenses detected, or only non-license legal texts such as CLAs), or `non_permissive`. The permissive allowlist follows the [Blue Oak Council](https://blueoakcouncil.org/list) list plus licenses categorized as *Permissive* or *Public Domain* by [ScanCode](https://scancode-licensedb.aboutcode.org/). **Files classified as `non_permissive` are excluded from both released datasets.**2873. **Near-deduplication.** MinHash signatures (256 permutations, 5-grams, min. 5 tokens) are indexed with LSH; candidate pairs with estimated Jaccard similarity ≥ **0.7** are grouped into duplicate clusters via connected components. Unlike v2, deduplication runs across all languages at once (language-agnostic) and candidate pairs are verified against their Jaccard estimate to eliminate false positives. For `stack-v3-train`, a single **representative** file per cluster is retained, chosen by (in order) highest stars, highest forks, permissive license, earliest repository creation.2884. **Quality heuristics** (applied to `stack-v3-train`). Same filters as the StarCoder2 pipeline, with minor adjustments to language lists to accommodate new or updated languages detected by the newer `go-enry` release:289   * Drop files with `<25%` alphabetic characters (alphanumeric for Assembly).290   * Drop files with average line length `>100` or maximum line length `>1000` (exempting data/markup formats such as Text, JSON, XML, HTML, Markdown, Roff, TeX).291   * Drop files with `>100,000` lines and cap several data/config formats (such as JSON, Yaml,...) at `512` lines.292   * Drop files with giant base64/hex/unicode-escape data blobs, LFS pointers, and files with auto-generation markers in the first 5 lines.293   * For HTML, keep only files whose visible text is ≥ 100 characters and ≥ 20% of the content.294   * Drop a curated blocklist of non-code languages/extensions.295   * Cap the largest markup/config languages (XML, HTML, JSON, JavaScript) at a fixed byte budget each, and probabilistically downsample any repository whose total size exceeds **1.5 GB** to fit Arrow/Parquet limits.2965. **PII redaction** (applied to `stack-v3-train`). Same model and filtering as v2. File contents are scanned with [`StarPII`](https://huggingface.co/bigcode/starpii). Detected emails, keys, names, passwords, and IP addresses are replaced with placeholder tokens (`<EMAIL>`, `<KEY>`, `<NAME>`, `<PASSWORD>`, and synthetic private IPs).2976. **Jupyter notebooks** (applied to `stack-v3-train`). `.ipynb` files are processed separately: cell **outputs, inline images, and volatile metadata are stripped**, and the notebook is kept as structured JSON. They pass through the same license, dedup, and PII steps. `stack-v3-full` keeps the raw `.ipynb` contents as crawled — use it if you want to apply your own notebook processing.2987. **Grouping.** Files are grouped by repository so each row is a full repository, enabling repository-context pre-training.299 300Opt-out requests (see below) are applied before release.301 302#### Who are the source language producers?303 304The source code was written by GitHub users whose repositories were public at crawl time (up to August 7, 2025).305 306### Personal and sensitive information307 308`stack-v3-train` was scanned for PII (emails, keys, names, passwords, IP addresses) with the same model and heuristics as The Stack v2, and detected entities were replaced with placeholder tokens. Detection is imperfect: the released data may still contain sensitive information (emails, IP addresses, keys) that was previously published to public GitHub repositories.309 310Researchers should use only public, non-personal information for open-access research, and must not use any personal information for spamming or other harmful purposes.311 312### Opting out313 314Developers can request that their code be removed from The Stack v3. You can check whether your code is included via the ["Am I in The Stack?"](https://huggingface.co/spaces/HuggingFaceCode/in-the-stack) Space, and submit removal requests following the [opt-out instructions](https://github.com/bigcode-project/opt-out-v2). Repositories that opted out are removed from the dataset before each patch release.315 316## Considerations for using the data317 318### Social impact319 320The Stack v3 is released in the spirit of open science, to increase the accessibility, reproducibility, and transparency of code-LLM research. Code LLMs can help people of diverse backgrounds write better software, but they also carry risks such as over-reliance on generated code, generation of insecure or malicious code, and effects on the software-development labor market.321 322### Discussion of biases323 324Widely used languages (e.g. C, JavaScript, Python) are heavily overrepresented relative to niche languages, and some languages are far less likely to be permissively licensed than others, biasing their representation. Comments in code may contain harmful or offensive language. The majority of natural language in the code is English.325 326### Other known limitations327 328* License attribution is only as accurate as ScanCode and the GitHub metadata; report any mistakes for review.329* The dataset may contain malicious code, and models trained on it could be used to generate malware.330* Because content is a single default-branch snapshot (as of August 7, 2025), it does not capture development history or non-default branches.331 332## Additional information333 334### Licensing information335 336The dataset is released under the **Open Data Commons Attribution License (ODC-By) v1.0** [license](https://opendatacommons.org/licenses/by/1-0/).337 338The Stack v3 is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in The Stack v3 must abide by the terms of the original licenses, including attribution clauses where relevant; we provide provenance information (`repo_path`, `commit_id`, `detected_licenses`) for each file to facilitate this. 339 340The list of [SPDX license identifiers](https://spdx.org/licenses/) included in the dataset can be found [here](stats/full/stats_by_license.json).341 342### Citation information343 344```bibtex345@misc{lozhkov2026stack-v3,346    author       = {Lozhkov, Anton and Larcher, Hugo and Morlon, Mathieu and Ben Allal, Loubna and von Werra, Leandro},347    title        = {The Stack v3: The Largest Open Code Dataset},348    year         = 2026,349    url          = {https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train},350    publisher    = {Hugging Face}351}352```353 354The Stack v3 builds on the methodology of The Stack v2 / StarCoder2:355 356```bibtex357@misc{lozhkov2024starcoder,358      title={StarCoder 2 and The Stack v2: The Next Generation},359      author={Anton Lozhkov and Raymond Li and Loubna Ben Allal and Federico Cassano and Joel Lamy-Poirier and Nouamane Tazi and Ao Tang and Dmytro Pykhtar and Jiawei Liu and Yuxiang Wei and Tianyang Liu and Max Tian and Denis Kocetkov and Arthur Zucker and Younes Belkada and Zijian Wang and Qian Liu and Dmitry Abulkhanov and Indraneil Paul and Zhuang Li and Wen-Ding Li and Megan Risdal and Jia Li and Jian Zhu and Terry Yue Zhuo and Evgenii Zheltonozhskii and Nii Osae Osae Dade and Wenhao Yu and Lucas Krauß and Naman Jain and Yixuan Su and Xuanli He and Manan Dey and Edoardo Abati and Yekun Chai and Niklas Muennighoff and Xiangru Tang and Muhtasham Oblokulov and Christopher Akiki and Marc Marone and Chenghao Mou and Mayank Mishra and Alex Gu and Binyuan Hui and Tri Dao and Armel Zebaze and Olivier Dehaene and Nicolas Patry and Canwen Xu and Julian McAuley and Han Hu and Torsten Scholak and Sebastien Paquet and Jennifer Robinson and Carolyn Jane Anderson and Nicolas Chapados and Mostofa Patwary and Nima Tajbakhsh and Yacine Jernite and Carlos Muñoz Ferrandis and Lingming Zhang and Sean Hughes and Thomas Wolf and Arjun Guha and Leandro von Werra and Harm de Vries},360      year={2024},361      eprint={2402.19173},362      archivePrefix={arXiv},363      primaryClass={cs.SE}364}365```366 367### Acknowledgements368 369This dataset is derived from source code hosted on GitHub. We thank the developers who make their work publicly available, and the [BigCode](https://www.bigcode-project.org/) community, whose work on dataset filters, PII detection, and open code dataset governance informed much of this pipeline.370