datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
starcoderdata
StarCoder Training Dataset
Dataset description
This is the dataset used for training StarCoder and StarCoderBase. It contains 783GB of code in 86 programming languages, and includes 54GB GitHub Issues + 13GB Jupyter notebooks in scripts and text-code pairs,
and 32GB of GitHub commits, which is approximately 250 Billion tokens.
Dataset creation
The creation and filtering of The Stack is explained in the original dataset, we additionally decontaminate and… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/starcoderdata.starcoderdata-python-edu-lang-score
Dataset Card for Starcoder Data with Python Education and Language Scores
Dataset Summary
The starcoderdata-python-edu-lang-score dataset contains the Python subset of the starcoderdata dataset. It augments the existing Python subset with features that assess the educational quality of code and classify the language of code comments. This dataset was created for high-quality Python education and language-based training, with a primary focus on facilitating models that can… See the full description on the dataset page: https://huggingface.co/datasets/JanSchTech/starcoderdata-python-edu-lang-score.starcoderdata-python-edu
starcoderdata-python-edu
StarCoder Training Dataset Cleaned and Scored
Dataset Details
Dataset Description
This dataset is a filtered version of StarCoder Training Dataset
that has been scored with the python-edu-scorer.
Dataset Sources
Repository: https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce15966
Paper: SmolLM - blazingly fast and remarkably powerful
Citation
@misc{allal2024SmolLM,
title={SmolLM… See the full description on the dataset page: https://huggingface.co/datasets/jon-tow/starcoderdata-python-edu.starcoderdata-gpt2starcoderdata-apisstarcoderdata_100star_py
Dataset Card for "starcoderdata_100star_py"
More Information needed
starcoderdata_100star_spy
Dataset Card for "starcoderdata_100star_spy"
More Information needed
starcoderdata_0.003
Dataset Card for "starcoderdata_0.003"
More Information needed
starcoderdata_ngram_10starcoderdataA clone of 'bigcode/starcoderdata' with 120k training rows and 30k validation rows
dataset_info:
features:
- name: max_stars_repo_path
dtype: string
- name: max_stars_repo_name
dtype: string
- name: max_stars_count
dtype: int64
- name: id
dtype: string
- name: content
dtype: string
splits:
- name: train
num_bytes: 771306855.2
num_examples: 120000
- name: validation
num_bytes: 192826713.8
num_examples: 30000
download_size:… See the full description on the dataset page: https://huggingface.co/datasets/NLTuan/starcoderdata.starcoderdata-typescript-ctx-6144starcoderdata-javascript-ctx-8192starcoderdata_100star_py_stripped_v1starcoderdata-go-ctx-4096starcoderdata-javascript-ctx-6144starcoderdata-javascript-ctx-4096starcoderdata-typescript-ctx-2048starcoderdata_py_smol
Dataset Card for "starcoderdata_py_smol"
More Information needed
starcoderdata-typescript-ctx-8192starcoderdata-javascript-ctx-2048starcoderdata-typescript-ctx-4096starcoderdata-java-ctx-4096starcoderdata-java-ctx-6144starcoderdata-rust-ctx-8192starcoderdata-rust-ctx-2048starcoderdata-rust-ctx-6144starcoderdata-python-ctx-4096starcoderdata-rust-ctx-4096starcoderdata-c-ctx-4096starcoderdata-python-ctx-2048