datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
conventional-commits
Git Diff → Conventional Commit Messages
A dataset of 12,433 (git diff, commit message) pairs scraped from real open-source TypeScript repositories, filtered for quality and formatted for fine-tuning small language models to generate Conventional Commits.
Built as part of a project to fine-tune a local LLM to write commit messages as a prepare-commit-msg git hook. Full write-up: eliotbas.com/projects/commits-fine-tuning
Dataset details
Source… See the full description on the dataset page: https://huggingface.co/datasets/Elib27/conventional-commits.github-commits-diff-dedup-pjjs-april
Deduplicated Commits
Deduplicated based on diff:
content = '\n'.join(difflib.unified_diff(
old_content.splitlines(keepends=True),
new_content.splitlines(keepends=True),
n=5
))
Parameters:
Minimum ngram size: 5
MinHash ngram size: 5
MinHash threshold: 0.8
git-commits-merged
Themis-Git-Commits-Merged
Overview
Themis-Git-Commits-Merged is a large-scale dataset of ~3.98M single-file code commits from permissively licensed GitHub repositories that have been cross-referenced with GHTorrent pull request data to retain only commits that are part of successfully merged, non-reverted pull requests. This provides implicit human validation of each code change — a merge decision by project maintainers confirms the intent and quality of… See the full description on the dataset page: https://huggingface.co/datasets/project-themis/git-commits-merged.commits-codegeex
Dataset Card for "commits-codegeex"
More Information needed
cvevc_commits
Dataset Card for Dataset Name
This dataset card aims to be a base template for new datasets. It has been generated using this raw template.
Dataset Details
Dataset Description
Curated by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Language(s) (NLP): [More Information Needed]
License: [More Information Needed]
Dataset Sources [optional]
Repository: [More… See the full description on the dataset page: https://huggingface.co/datasets/andstor/cvevc_commits.commits-pjj-diff
Dataset Card for "commits-pjj-diff"
More Information needed
tangled-ccs-commits
Detecting Multiple Semantic Concerns in Tangled Code Commits using Small Language Models
This dataset contains commit data for training and evaluating models on software engineering tasks, specifically focusing on identifying and separating concerns in multi-concern commits.
Every tangled (multi-concern) commit in this dataset is composed exclusively of atomic commits from a single repository — resolving a structural weakness in earlier cross-repo tangles (which were trivially… See the full description on the dataset page: https://huggingface.co/datasets/Berom0227/tangled-ccs-commits.pretrain-commits-v2-mix-long-contextNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
12,571,681,749 (12.6B)
Trainable tokens
4,460,160,435 (4.5B)
Documents
992,475
Shards
327
UTF-8 bytes
49,288,867,997
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum, sizes… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-commits-v2-mix-long-context.pretrain-commits-v2-mixNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
79,318,557,379 (79.3B)
Trainable tokens
28,772,968,648 (28.8B)
Documents
31,431,846
Shards
696
UTF-8 bytes
310,412,170,445
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-commits-v2-mix.commits_sample_filesThis is a sample of GitHub commits and files reconstructed from the Software Heritage dataset.
It contains the latest 1024 commits in all of pytorch/* and huggingface/* repos.
The tables are split to avoid an explosion of rows (lots of repeated files between commits), so you will need to pre-filter the commits before adding the file contents.
Table descriptions:
1. commits
The commit message table. Join it with commit_filepath on commits.directory_id == commit_filepath.directory_id… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/commits_sample_files.commits-pjj-2048
Dataset Card for "commits-pjj-2048"
More Information needed
ccs_commits_with_bodyvazi-corpus-backup_remove-after-main-commits-properlySWE-bench_oracle-parsed-commitscockroach-commitsvaani-gujarati-sft-data
Vaani — Gujarati SFT & Eval Data
Final datasets for the Vaani 110M Gujarati medical SLM.
File
Rows
Purpose
sft_v4.jsonl
~150k
Instruction SFT: general instructions + format skills (JSON, extraction, exact-count lists)
medical_sft_v3.jsonl
~114k
Medical SFT: closed-book MedMCQA-gu + raw-grounded + redacted-grounded + 30% general mix
medmcqa_gu_val.jsonl
4,183
Held-out MedMCQA-gu validation split (eval only; disjoint from SFT)
The pretraining corpus is… See the full description on the dataset page: https://huggingface.co/datasets/pratham-commits/vaani-gujarati-sft-data.SWE-bench_Verified_oracle-parsed_commits_32k_5git-commits
Dataset: dataset.jsonl
Auto-labeled commit dataset scraped from GitHub repositories. Each line is a JSON object representing one commit with extracted features and an inferred label.
Features
Field
Type
Description
Stats
text
string
Commit message first line, conventional prefix stripped
—
files_count
int
Number of files changed
mean 4.3, median 1, max 300
additions
int
Lines added
mean 88, median 6, max 187K
deletions
int
Lines deleted
mean 172… See the full description on the dataset page: https://huggingface.co/datasets/akaruineko/git-commits.guanaco-commits
Dataset Card for "guanaco-commits"
More Information needed
github-commits-diff-dedup-pjjs-april-git_diffs-subset-3861806-processed-subsetSWE-bench_Verified_oracle-parsed_commits_32k_2SWE-bench_oracle-parsed-commits_100vulnerable-functions-and-commits_cvefixes-2022
vulnerable-functions-and-commits_cvefixes-2022
Contains vulnerable functions and commits from the CVEFixes SQLite database.
commits-8192SWE-bench_Verified_oracle-parsed_commits_32k_100openbsd-commits-alpaca
OpenBSD Commit History — Alpaca Format (v1)
Fine-tuning dataset derived from the full commit history of the
OpenBSD src repository, structured
for instruction fine-tuning in Alpaca format.
Task: given a unified diff, generate the commit message.
Dataset Summary
Field
Value
Examples
103,383
Size
~202 MB
Format
Alpaca JSONL
Date range
2000-01-01 → present
Source
openbsd/src (GitHub mirror)
License
ISC
Format
Each line is a… See the full description on the dataset page: https://huggingface.co/datasets/ajsbsd/openbsd-commits-alpaca.bigvul_devign_cvefixes_neuralsentry_commits
Dataset Card for "bigvul_devign_cvefixes_neuralsentry_commits"
More Information needed
2016-Request-for-Commits-Transcripts
2016 Request for Commits Transcripts
Complete transcripts from the 2016 episodes of the Request for Commits podcast.
Generated from this GitHub repository.
gujarati-autoscientist-datasetagent-commits-raw
AI Coding-Agent Commits on GitHub
This dataset documents commits associated with four AI coding agents: Claude, OpenAI Codex, GitHub Copilot and Cursor. It contains 1,853,915 commit records across 444,055 GitHub repositories, with 220,753 identifiable GitHub user accounts recorded as commit authors. Messages and author identities are in commits; repository metadata, file changes and patch text are available in separate tables.
Dataset
Agent
Commit records… See the full description on the dataset page: https://huggingface.co/datasets/ASSERT-KTH/agent-commits-raw.