datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
starcoderdata
StarCoder Training Dataset
Dataset description
This is the dataset used for training StarCoder and StarCoderBase. It contains 783GB of code in 86 programming languages, and includes 54GB GitHub Issues + 13GB Jupyter notebooks in scripts and text-code pairs,
and 32GB of GitHub commits, which is approximately 250 Billion tokens.
Dataset creation
The creation and filtering of The Stack is explained in the original dataset, we additionally decontaminate and… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/starcoderdata.starcoderdata-python-edu-lang-score
Dataset Card for Starcoder Data with Python Education and Language Scores
Dataset Summary
The starcoderdata-python-edu-lang-score dataset contains the Python subset of the starcoderdata dataset. It augments the existing Python subset with features that assess the educational quality of code and classify the language of code comments. This dataset was created for high-quality Python education and language-based training, with a primary focus on facilitating models that can… See the full description on the dataset page: https://huggingface.co/datasets/JanSchTech/starcoderdata-python-edu-lang-score.starcoderdata-python-edu
starcoderdata-python-edu
StarCoder Training Dataset Cleaned and Scored
Dataset Details
Dataset Description
This dataset is a filtered version of StarCoder Training Dataset
that has been scored with the python-edu-scorer.
Dataset Sources
Repository: https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce15966
Paper: SmolLM - blazingly fast and remarkably powerful
Citation
@misc{allal2024SmolLM,
title={SmolLM… See the full description on the dataset page: https://huggingface.co/datasets/jon-tow/starcoderdata-python-edu.marin-starcoderdata_visual-basicstarcoderdata-gpt2starcoderdatamarin-starcoderdata_fortranstarcoderdatastarcoderdata-repoCombines the data from starcoderdata and removes any repos with <= 10 stars.
marin-starcoderdata_verilogmarin-starcoderdata_elmmarin-starcoderdata_vhdlstarcoderdata-samplemarin-starcoderdata_glslmarin-starcoderdata_perlmarin-starcoderdata_yaccstarcoderdata-apismarin-starcoderdata_cssmarin-starcoderdata_dockerfilemarin-starcoderdata_groovymarin-starcoderdata_thriftmarin-starcoderdata_racketmarin-starcoderdata_rubymarin-starcoderdata_tcshmarin-starcoderdata_texmarin-starcoderdata_gogrouped-starcoderdata-mergedmarin-starcoderdata_powershellmarin-starcoderdata_yamlmarin-starcoderdata_standard-ml