datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
starcoderdata
StarCoder Training Dataset
Dataset description
This is the dataset used for training StarCoder and StarCoderBase. It contains 783GB of code in 86 programming languages, and includes 54GB GitHub Issues + 13GB Jupyter notebooks in scripts and text-code pairs,
and 32GB of GitHub commits, which is approximately 250 Billion tokens.
Dataset creation
The creation and filtering of The Stack is explained in the original dataset, we additionally decontaminate and… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/starcoderdata.starcoder2data-extras
StarCoder2 Extras
This is the dataset of extra sources (besides Stack v2 code data) used to train the StarCoder2 family of models. It contains the following subsets:
Kaggle (kaggle): Kaggle notebooks from Meta-Kaggle-Code dataset, converted to scripts and prefixed with information on the Kaggle datasets used in the notebook. The file headers have a similar format to Jupyter Structured but the code content is only one single script.
StackOverflow (stackoverflow): stackoverflow… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/starcoder2data-extras.starcoderdata-python-edu-lang-score
Dataset Card for Starcoder Data with Python Education and Language Scores
Dataset Summary
The starcoderdata-python-edu-lang-score dataset contains the Python subset of the starcoderdata dataset. It augments the existing Python subset with features that assess the educational quality of code and classify the language of code comments. This dataset was created for high-quality Python education and language-based training, with a primary focus on facilitating models that can… See the full description on the dataset page: https://huggingface.co/datasets/JanSchTech/starcoderdata-python-edu-lang-score.stack-v2-starcoder2-3bstarcoderdata-python-edu
starcoderdata-python-edu
StarCoder Training Dataset Cleaned and Scored
Dataset Details
Dataset Description
This dataset is a filtered version of StarCoder Training Dataset
that has been scored with the python-edu-scorer.
Dataset Sources
Repository: https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce15966
Paper: SmolLM - blazingly fast and remarkably powerful
Citation
@misc{allal2024SmolLM,
title={SmolLM… See the full description on the dataset page: https://huggingface.co/datasets/jon-tow/starcoderdata-python-edu.starcoder_cleanedThis is starcoderdata, but with leading boilerplate text/license text removed, and with short sequences filtered out. It also removes the extra tags at the beginning of some of the files, like <reponame>.
starcoderdata-gpt2self-instruct-starcoder
Self-instruct-starcoder
Summary
Self-instruct-starcoder is a dataset that was generated by prompting starcoder to generate new instructions based on some human-written seed instructions.
The underlying process is explained in the paper self-instruct. This algorithm gave birth to famous machine generated
datasets such as Alpaca and Code Alpaca which are two datasets
obtained by prompting OpenAI text-davinci-003 engine.
Our approach
While our method is… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/self-instruct-starcoder.starcoder_filtered
Dataset Card for "starcoder_filtered"
A version of the starcoder dataset filtered based on data quality. Data was labeled with a rater model, and low-ranking rows were removed.
starcoder_labeled
Dataset Card for "starcoder_labeled"
Starcoder data, with several popular languages selected, short sequences filtered out, then labeled based on learning quality (educational value) and code quality.
A good heuristic is to take anything with >.5 code quality and >.3 learning quality. But you may want to vary the thresholds by language, depending on your target task.
starcoder-numpy-pandasstarcoder-chemistrystarcoderdata-apisstarcoder
Starcoder Dataset (The Stack - Sub-sampled)
This dataset is derived from the "Starcoder" version of The Stack, a 6.4 TB dataset of permissively licensed source code in 384 programming languages.
This repository contains the data organized into subsets, one for each programming language or data type.
How to Use
You can load any language-specific subset of the data using the datasets library. You must specify the name parameter with the desired language.
For example… See the full description on the dataset page: https://huggingface.co/datasets/Sam-Shin/starcoder.starcoder-apis-2starcoder-apis-1starcoder-python5b5b gpt2 tokens
starcoder-apis-0starcoder2-documentation
Dataset Card
This dataset is the code documenation dataset used in StarCoder2 pre-training, and it is also part of the-stack-v2-train-extras descried in the paper.
Dataset Details
Overview
This dataset comprises a comprehensive collection of crawled documentation and code-related resources sourced from various package manager platforms and programming language documentation sites. It focuses on popular libraries, free programming books, and other relevant… See the full description on the dataset page: https://huggingface.co/datasets/SivilTaram/starcoder2-documentation.common_starcoder
Common Starcoder dataset
This dataset is generated from bigcode/starcoderdata.
Total GPT2 Tokens: 4,649,163,171
Generation Process
We filtered the original dataset with common language: C, Cpp, Java, Python and JSON.
We removed some columns for mixing up with other dataset: "id", "max_stars_repo_path", "max_stars_repo_name"
After removing the irrelevant fields, we shuffle the dataset with random seed=42.
We filtered the data on "max_stars_count" > 300 and shuffle again.… See the full description on the dataset page: https://huggingface.co/datasets/skymizer/common_starcoder.starcoder-finetune
Dataset Card for "starcoder-finetune"
More Information needed
starcoderdata_100star_py
Dataset Card for "starcoderdata_100star_py"
More Information needed
starcoder-curated
StarCoderData Curated
A curated subset of StarCoderData
optimised for training a 500M parameter model focused on structured data output
(JSON generation, function calling, schema compliance).
Dataset Summary
Total code files: 5,203,508
Total tokens: 3.9B (target: 3.5B)
Classifier-scored files: 1,553,596 (1.7B tokens)
Non-classified files: 3,649,912 (2.2B tokens) — filtered by heuristics, not the classifier
Source: bigcode/starcoderdata
Classifier:… See the full description on the dataset page: https://huggingface.co/datasets/mdonigian/starcoder-curated.starcoderdata_100star_spy
Dataset Card for "starcoderdata_100star_spy"
More Information needed
stack-dedup-python-testgen-starcoder-filter-v2
MultiPL-T Python Sources
Citation
If you use this dataset we request that you cite our work:
@misc{cassano:multipl-t,
title={Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs},
author={Federico Cassano and John Gouwar and Francesca Lucchetti and Claire Schlesinger and Anders Freeman and Carolyn Jane Anderson and Molly Q Feldman and Michael Greenberg and Abhinav Jangda and Arjun Guha},
year={2024}… See the full description on the dataset page: https://huggingface.co/datasets/nuprl/stack-dedup-python-testgen-starcoder-filter-v2.starcoder-filtered-apisstarcoderdata_0.003
Dataset Card for "starcoderdata_0.003"
More Information needed
starcoder-bpb-3kjailbreaks_dataset_with_perplexity_bigcode_starcoder2-3b_bigcode_starcoder2-7bStarCoder_tokenized