datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
github_archive
GitHub Archive
Description
According to GitHub’s terms of service, issues and pull request descriptions—along with the their comments—inherit the license of their associated repository.
To collect this data, we used the GitHub Archive’s public BigQuery table of events to extracted all issue, pull request, and comment events since 2011 and aggregated them into threads.
The table appeared to be missing “edit” events so the text from each comment is the original from when… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/github_archive.github-jupyter
GitHub Jupyter Dataset
Dataset Description
The dataset was extracted from Jupyter Notebooks on BigQuery.
Licenses
Each example has the license of its associated repository. There are in total 15 licenses:
[
'mit',
'apache-2.0',
'gpl-3.0',
'gpl-2.0',
'bsd-3-clause',
'agpl-3.0',
'lgpl-3.0',
'lgpl-2.1',
'bsd-2-clause',
'cc0-1.0',
'epl-1.0',
'mpl-2.0',
'unlicense',
'isc',
'artistic-2.0'
]
github_archive_filtered
GitHub Archive
Description
According to GitHub’s terms of service, issues and pull request descriptions—along with their comments—inherit the license of their associated repository.
To collect this data, we used the GitHub Archive’s public BigQuery table of events to extract all issue, pull request, and comment events since 2011 and aggregated them into threads.
The table appeared to be missing “edit” events so the text from each comment is the original from when it was… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/github_archive_filtered.GitHub-CC0
Public Domain GitHub Repositories Dataset
This dataset contains metadata and source code of 9,000 public domain (cc0 or unlicense) licensed GitHub repositories that have more than 25 stars.
The dataset was created by scraping the GitHub API and downloading the repositories, so long as they are under 100mb.
The dataset can be used for various natural language processing and software engineering tasks, such as code summarization, code generation, code search, code analysis, etc.… See the full description on the dataset page: https://huggingface.co/datasets/KoalaAI/GitHub-CC0.1c_githubgithub-source-code-dataset
Github Source Code Dataset
Complete source code from Agnuxo projects.
Part of the Agnuxo Ecosystem by Francisco Angulo de Lafuente.
github-docs
GitHub Docs Corpus
A dataset containing only information from GitHub — the official
github/docs repository, i.e. the source of docs.github.com.
Dataset Structure
Files: data/train.jsonl
Format: JSONL, one chunk per line
Columns: text (cleaned doc chunk), metadata (source, title)
Rows: 3,336
Composition
Source: github/docs (main branch), content/ tree only — 3,734
Markdown files covering GitHub features, workflows, webhooks, REST/GraphQL
API docs… See the full description on the dataset page: https://huggingface.co/datasets/Nanthasit/github-docs.github-mini
github-mini
A dataset consisting of source code from GitHub repositories.
Dataset Curation
Source: GitHub repositories with 500 to 99,999+ stars.
Licenses: Filtered for MIT and Apache-2.0.
Focus: Source code across multiple programming languages.
Dataset Structure
Each record in the dataset contains the following fields:
repo_full_name: The full name of the repository (owner/name).
repo_url: Direct link to the GitHub repository.
stars: Number of stars at the… See the full description on the dataset page: https://huggingface.co/datasets/lumasik/github-mini.github-issue-pr-resolution-10k
GitHub Issue→PR Resolution Dataset — 10K Sample
This is a 10,000-record sample. The full dataset (~354,000 records) is available for purchase. Contact andrewcampi456@gmail.com for licensing details.
What is this?
A large-scale dataset of resolved GitHub issues paired with their linked pull requests, code diffs, commits, and review discussions, all from production-quality Python repositories.
Each record captures the complete lifecycle of a software fix:
Issue… See the full description on the dataset page: https://huggingface.co/datasets/acampi23/github-issue-pr-resolution-10k.github-ai-projects-dataset
GitHub Code Instruction Dataset for LLM Fine-Tuning
Dataset Description
This dataset contains high-quality code instruction examples extracted from popular GitHub repositories focused on LLMs, LangChain, FastAPI, Django, and Transformers. It is designed for supervised fine-tuning of large language models (LLMs) for code generation, completion, and documentation tasks.
Dataset Structure
The dataset is split into three parts:
Train: 80% of examples for model… See the full description on the dataset page: https://huggingface.co/datasets/pranav-pvnn/github-ai-projects-dataset.