datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
dataset-the-stack-v2-dedup-sub
The Stack v2 Subset with File Contents (Python, Java, JavaScript, C, C++)
TempestTeam/dataset-the-stack-v2-dedup-sub
Dataset Summary
This dataset is a language-filtered and self-contained subset of bigcode/the-stack-v2-dedup, part
of the BigCode Project.
It contains only files written in the following programming languages:
Python 🐍
Java ☕
JavaScript 📜
C ⚙️
C++ ⚙️
Unlike the original dataset, which only includes metadata and Software Heritage IDs, this subset includes… See the full description on the dataset page: https://huggingface.co/datasets/Cyrile/dataset-the-stack-v2-dedup-sub.the-stack-v2-dedup
The Stack v2
The dataset consists of 4 versions:
bigcode/the-stack-v2: the full "The Stack v2" dataset
bigcode/the-stack-v2-dedup: based on the bigcode/the-stack-v2 but further near-deduplicated <-- you are here
bigcode/the-stack-v2-train-full-ids: based on the bigcode/the-stack-v2-dedup dataset but further filtered with heuristics and spanning 600+ programming languages. The data is grouped into repositories.
bigcode/the-stack-v2-train-smol-ids: based on the… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/the-stack-v2-dedup.the-stack-v2the-stack-v2-pythonthe-stack-v2-cthe-stack-v2-new-pythonthe-stack-v2-new-cthe-stack-v2-new-cppthe-stack-v2-train-smol-ids-updatedUpdate on The Stack V2 dataset: https://huggingface.co/datasets/bigcode/the-stack-v2-train-smol-ids
All repos from original dataset are parsed with Github API and re-downloaded,
so respective updates are kept, metadata is updated. This took 10+ days to process due to GraphQL limits.
Filtering rules
Removed repos with no update in the last 6 years (no updates since September 2019)
Removed files with a single line
Removed repos with a single file
Removed repos with more than 99%… See the full description on the dataset page: https://huggingface.co/datasets/thepowerfuldeez/the-stack-v2-train-smol-ids-updated.the-stack-v2-train-smol-ids
The Stack v2
The dataset consists of 4 versions:
bigcode/the-stack-v2: the full "The Stack v2" dataset
bigcode/the-stack-v2-dedup: based on the bigcode/the-stack-v2 but further near-deduplicated
bigcode/the-stack-v2-train-full-ids: based on the bigcode/the-stack-v2-dedup dataset but further filtered with heuristics and spanning 600+ programming languages. The data is grouped into repositories.
bigcode/the-stack-v2-train-smol-ids: based on the bigcode/the-stack-v2-dedup… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/the-stack-v2-train-smol-ids.the-stack-v2-train-full-ids
The Stack v2
The dataset consists of 4 versions:
bigcode/the-stack-v2: the full "The Stack v2" dataset
bigcode/the-stack-v2-dedup: based on the bigcode/the-stack-v2 but further near-deduplicated
bigcode/the-stack-v2-train-full-ids: based on the bigcode/the-stack-v2-dedup dataset but further filtered with heuristics and spanning 600+ programming languages. The data is grouped into repositories. <-- you are here
bigcode/the-stack-v2-train-smol-ids: based on the… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/the-stack-v2-train-full-ids.the-stack-v2-cppthe-stack-v2-processedthe-stack-v2-dedup-filtered-500-stars-100-forks-contentsthe-stack-v2-blamedthe-stack-v2-filteredthe-stack-v2-train-xsmol-content
The Stack v2 — 12-Language Resolved Content
This dataset provides resolved file content for twelve programming languages,
derived from the repository/file identifiers published in
bigcode/the-stack-v2-train-full-ids.
The upstream dataset ships identifiers only — each file is a pointer into the
Software Heritage archive. Here, those
identifiers have been resolved to their actual source text so the content is
directly usable, with the upstream metadata carried through unchanged.… See the full description on the dataset page: https://huggingface.co/datasets/handwoven8588/the-stack-v2-train-xsmol-content.the-stack-v2-filtered-cthe-stack-v2-pythonthe-stack-v2-filtered-cppthe-stack-v2-filtered-pythonthe-stack-v2-javaThe_Stack_Processed-v2
🔥 The Stack Processed V2
A curated, balanced, and ML-optimized multi-language programming dataset
🎯 Why Choose This Dataset?
A meticulously curated version of "The Stack" optimized for training robust multi-language code models. Perfect balance between quality, diversity, and usability.
✨ Key Advantages:
🎯 Perfect Balance: ~10,000 files per major programming language
⚡ Training-Ready: Parquet format optimized for ML workflows
🏆 Superior Quality: 91.3% syntax… See the full description on the dataset page: https://huggingface.co/datasets/vinsblack/The_Stack_Processed-v2.the-stack-v2-blamed2the-stack-v2-dedup-sql-annotateThis dataset focuses the SQL corpus of the Stack v2 dedup dataset and provides annotations to each SQL file therein.
sqlparse is used to parse the SQL code, then count keywords and symbols.
Below are the annotation columns.
Column Name
Column Description
Keyword.DML
Data Manipulation Language commands for modifying database records, e.g. SELECT, INSERT, UPDATE, DELETE, COMMIT, MERGE, ROLLBACK.
Keyword.DDL
Data Definition Language commands for defining or altering database… See the full description on the dataset page: https://huggingface.co/datasets/onekq-ai/the-stack-v2-dedup-sql-annotate.the-stack-v2-blamed-pythonthe-stack-v2-new-javathe-stack-v2-blamed-cppthe-stack-v2-dedup285kthe-stack-v2-filtered-java