datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
starcoderdata
StarCoder Training Dataset
Dataset description
This is the dataset used for training StarCoder and StarCoderBase. It contains 783GB of code in 86 programming languages, and includes 54GB GitHub Issues + 13GB Jupyter notebooks in scripts and text-code pairs,
and 32GB of GitHub commits, which is approximately 250 Billion tokens.
Dataset creation
The creation and filtering of The Stack is explained in the original dataset, we additionally decontaminate and… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/starcoderdata.starcoder2data-extras
StarCoder2 Extras
This is the dataset of extra sources (besides Stack v2 code data) used to train the StarCoder2 family of models. It contains the following subsets:
Kaggle (kaggle): Kaggle notebooks from Meta-Kaggle-Code dataset, converted to scripts and prefixed with information on the Kaggle datasets used in the notebook. The file headers have a similar format to Jupyter Structured but the code content is only one single script.
StackOverflow (stackoverflow): stackoverflow… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/starcoder2data-extras.mosaic-starcoder-filtered
Mosaic format for filtered starcoder dataset to train Malaysian LLM
This repository is to store dataset shards using mosaic format.
prepared at https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-starcoder.ipynb
using tokenizer https://huggingface.co/malaysia-ai/bpe-tokenizer
4096 context length.
how-to
git clone,
git lfs clone https://huggingface.co/datasets/malaysia-ai/mosaic-starcoder-filtered
load it,
from streaming import… See the full description on the dataset page: https://huggingface.co/datasets/malaysia-ai/mosaic-starcoder-filtered.starcoderdata-python-edu-lang-score
Dataset Card for Starcoder Data with Python Education and Language Scores
Dataset Summary
The starcoderdata-python-edu-lang-score dataset contains the Python subset of the starcoderdata dataset. It augments the existing Python subset with features that assess the educational quality of code and classify the language of code comments. This dataset was created for high-quality Python education and language-based training, with a primary focus on facilitating models that can… See the full description on the dataset page: https://huggingface.co/datasets/JanSchTech/starcoderdata-python-edu-lang-score.stack-v2-starcoder2-3bstarcoderdata-python-edu
starcoderdata-python-edu
StarCoder Training Dataset Cleaned and Scored
Dataset Details
Dataset Description
This dataset is a filtered version of StarCoder Training Dataset
that has been scored with the python-edu-scorer.
Dataset Sources
Repository: https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce15966
Paper: SmolLM - blazingly fast and remarkably powerful
Citation
@misc{allal2024SmolLM,
title={SmolLM… See the full description on the dataset page: https://huggingface.co/datasets/jon-tow/starcoderdata-python-edu.starcoder_cleanedThis is starcoderdata, but with leading boilerplate text/license text removed, and with short sequences filtered out. It also removes the extra tags at the beginning of some of the files, like <reponame>.
starcoder-python-instruct
StarCoder-Python-Qwen-Instruct
Dataset Description
This dataset contains Python code samples paired with synthetically generated natural language instructions. It is designed for supervised fine-tuning of language models for code generation tasks. The dataset is derived from the Python subset of the bigcode/starcoderdata corpus, and the instructional text for each code sample was generated using the Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 model.
Creation… See the full description on the dataset page: https://huggingface.co/datasets/OLMo-Coding/starcoder-python-instruct.starcoder_mds_incrementalstarcoder2-instruct-assetsstarcoder_filtered
Dataset Card for "starcoder_filtered"
A version of the starcoder dataset filtered based on data quality. Data was labeled with a rater model, and low-ranking rows were removed.
marin-starcoderdata_visual-basicstarcoderdata-gpt2self-instruct-starcoder
Self-instruct-starcoder
Summary
Self-instruct-starcoder is a dataset that was generated by prompting starcoder to generate new instructions based on some human-written seed instructions.
The underlying process is explained in the paper self-instruct. This algorithm gave birth to famous machine generated
datasets such as Alpaca and Code Alpaca which are two datasets
obtained by prompting OpenAI text-davinci-003 engine.
Our approach
While our method is… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/self-instruct-starcoder.starcoderdatamarin-starcoderdata_fortranstarcoderdatastarcoder_labeled
Dataset Card for "starcoder_labeled"
Starcoder data, with several popular languages selected, short sequences filtered out, then labeled based on learning quality (educational value) and code quality.
A good heuristic is to take anything with >.5 code quality and >.3 learning quality. But you may want to vary the thresholds by language, depending on your target task.
starcoderdata-repoCombines the data from starcoderdata and removes any repos with <= 10 stars.
marin-starcoderdata_verilogmarin-starcoderdata_elmmarin-starcoderdata_vhdlstarcoderdata-samplestarcoder-numpy-pandasmarin-starcoderdata_glslmarin-starcoderdata_perlstarcoder
Starcoder Dataset (The Stack - Sub-sampled)
This dataset is derived from the "Starcoder" version of The Stack, a 6.4 TB dataset of permissively licensed source code in 384 programming languages.
This repository contains the data organized into subsets, one for each programming language or data type.
How to Use
You can load any language-specific subset of the data using the datasets library. You must specify the name parameter with the desired language.
For example… See the full description on the dataset page: https://huggingface.co/datasets/Sam-Shin/starcoder.starcoder-chemistrymarin-starcoderdata_yaccstarcoderdata-apis