datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
commit0commitbench_conventionalcommit-chronicle
📜 CommitChronicle 🔮
This is the dataset for commit message generation (and/or completion), introduced in the paper "From Commit Message Generation to History-Aware Commit Message Completion", ASE 2023.
Its key features:
large-scale and multilingual: contains 10.7M commits from 11.9k GitHub repositories in 20 programming languages;
diverse: avoids restrictive filtering on commit messages or commit diffs structure;
suitable for experiments with commit history: provides metadata… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/commit-chronicle.github-commits-diff-dedup-pjjs-april
Deduplicated Commits
Deduplicated based on diff:
content = '\n'.join(difflib.unified_diff(
old_content.splitlines(keepends=True),
new_content.splitlines(keepends=True),
n=5
))
Parameters:
Minimum ngram size: 5
MinHash ngram size: 5
MinHash threshold: 0.8
commit0_combinedgit-commits-merged
Themis-Git-Commits-Merged
Overview
Themis-Git-Commits-Merged is a large-scale dataset of ~3.98M single-file code commits from permissively licensed GitHub repositories that have been cross-referenced with GHTorrent pull request data to retain only commits that are part of successfully merged, non-reverted pull requests. This provides implicit human validation of each code change — a merge decision by project maintainers confirms the intent and quality of… See the full description on the dataset page: https://huggingface.co/datasets/project-themis/git-commits-merged.commitpackmetaGitHub metadata for https://huggingface.co/datasets/bigcode/commitpack
knesset-committees-chunkscommits-codegeex
Dataset Card for "commits-codegeex"
More Information needed
commit-chronicle
📜 CommitChronicle 🔮
This is the dataset for commit message generation (and/or completion), introduced in the paper "From Commit Message Generation to History-Aware Commit Message Completion", ASE 2023.
Its key features:
large-scale and multilingual: contains 10.7M commits from 11.9k GitHub repositories in 20 programming languages;
diverse: avoids restrictive filtering on commit messages or commit diffs structure;
suitable for experiments with commit history: provides metadata… See the full description on the dataset page: https://huggingface.co/datasets/Hirunima/commit-chronicle.cvevc_commits
Dataset Card for Dataset Name
This dataset card aims to be a base template for new datasets. It has been generated using this raw template.
Dataset Details
Dataset Description
Curated by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Language(s) (NLP): [More Information Needed]
License: [More Information Needed]
Dataset Sources [optional]
Repository: [More… See the full description on the dataset page: https://huggingface.co/datasets/andstor/cvevc_commits.commits-pjj-diff
Dataset Card for "commits-pjj-diff"
More Information needed
exp-pool-commit-code-dolma2-tokenized
Locus EXP Commit Code - Dolma 2 tokenized
Pretokenized experiment pool for reproducible proxy-training runs.
MANIFEST.json is the authoritative schema, provenance, checksums, and train/holdout assignment.
shards/shard-NNNNN/tokens.bin stores little-endian int32 token IDs.
offsets.bin stores little-endian int64 document boundaries.
index.parquet stores document IDs, offsets, and compact filter fields.
metadata.parquet stores complete source metadata and is downloaded only for… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/exp-pool-commit-code-dolma2-tokenized.lca-commit-message-generation
🏟️ Long Code Arena (Commit message generation)
This is the benchmark for the Commit message generation task as part of the
🏟️ Long Code Arena benchmark.
The dataset is a manually curated subset of the Python test set from the 🤗 CommitChronicle dataset, tailored for larger commits.
All the repositories are published under permissive licenses (MIT, Apache-2.0, and BSD-3-Clause). The datapoints can be removed upon request.
How-to
from datasets import load_dataset… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/lca-commit-message-generation.pretrain-commits-v2-mix-long-contextNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
12,571,681,749 (12.6B)
Trainable tokens
4,460,160,435 (4.5B)
Documents
992,475
Shards
327
UTF-8 bytes
49,288,867,997
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum, sizes… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-commits-v2-mix-long-context.SWE-bench_Verified_commit0pretrain-commits-v2-mixNormalized documents plus aligned Dolma-2 tokens and target masks.
Size
Tokens
79,318,557,379 (79.3B)
Trainable tokens
28,772,968,648 (28.8B)
Documents
31,431,846
Shards
696
UTF-8 bytes
310,412,170,445
Tokenizer
allenai/dolma2-tokenizer@5292e5d6c0f4
documents.parquet - document_id, text, part_ends, part_trainable,
must_not_split. The readable payload and the mask intent.
metadata.parquet - one text-free row per document: token span, source,
stratum… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/pretrain-commits-v2-mix.commits_sample_filesThis is a sample of GitHub commits and files reconstructed from the Software Heritage dataset.
It contains the latest 1024 commits in all of pytorch/* and huggingface/* repos.
The tables are split to avoid an explosion of rows (lots of repeated files between commits), so you will need to pre-filter the commits before adding the file contents.
Table descriptions:
1. commits
The commit message table. Join it with commit_filepath on commits.directory_id == commit_filepath.directory_id… See the full description on the dataset page: https://huggingface.co/datasets/bigcode/commits_sample_files.qmmit-open-source-agent-commit-index
Repository-Level Measurement of Self-Declared Coding-Agent Commit Signatures
Dataset release: 2026-09-18-v3.0Schema: 3.0.0
Release stamp: dataset 2026-09-18-v3.0 · ruleset sha256:b2e8889c66f72c18a61839f0bf1a9f77b5481ba2def044dba33c797b2f2bdcae · scanned 2026-09-16
Abstract
This dataset contains 2000 repository-level observations from public Git
repositories. Each observation estimates a lower bound on the proportion of
non-merge, non-infrastructure-bot commits… See the full description on the dataset page: https://huggingface.co/datasets/balrampandey/qmmit-open-source-agent-commit-index.commits-pjj-2048
Dataset Card for "commits-pjj-2048"
More Information needed
commitments-of-traders
Commitments of Traders
Who was long and who was short in every US futures market — and, for once,
when anyone could actually see it.
421 223 market-weeks · 211 071 point-in-time rows · 2 585 weekly releases ·
762 markets · 2010-01-05 to 2026-09-08
The pipeline lives in recipe/ at the same revision as the data.
See PIPELINE.md for the method.
The data is from Tuesday. It comes out on Friday.
A COT report is taken as of the close on Tuesday and published at 3:30… See the full description on the dataset page: https://huggingface.co/datasets/ZipLime/commitments-of-traders.processed-commit-diffs
List of repositories included in the dataset
Project
Language
Fetched Count
Url
Moby
Go
5 943
https://github.com/moby/moby
Rxjava
Java
516
https://github.com/RxJava/ReactiveX
Spring-framework
Java
2 529
https://github.com/spring-framework/spring-project
Chart.js
Javascript
641
https://github.com/Chart.js/chartjs
Three.js
Javascript
1 512https://github.com/three.js/mrdoob
Redux
Javascript
592
https://github.com/redux/reduxjs
React-native
Javascript
2 901… See the full description on the dataset page: https://huggingface.co/datasets/mamiksik/processed-commit-diffs.CommitPackFTcommit-chronicleThis is a filtered version of the JetBrains-Research/commit-chronicle dataset. It has been subsetted for the following languages:
[
"C", "C++", "Go", "Java", "Python", "Rust", "TypeScript"
]
Further filtering steps undertaken are:
Useless features have been removed and only the message and diff retained.
Only commits that modify a single file have been chosen.
Samples containing diffs longer than 1024 tokens (by the ObscuraCoder/Tokenizer tokenizer estimate) have been discarded.commitpack-ft-instruct-ratedThis is commitpack-ft-instruct, derived from Octocode's CommitPackFT, augmented with a quality analysis of the instruction-response pair by a local model. This did a pretty decent job of identifying pairs that obviously don't have enough context to know what change is being requested, or where the commit message does not match with the changes made.
Data files (yaml, plain text, json, etc.) were heavily downsampled in preparing this dataset to skew it more towards actual code work. All entries… See the full description on the dataset page: https://huggingface.co/datasets/chargoddard/commitpack-ft-instruct-rated.vazi-corpus-backup_remove-after-main-commits-properlysejm-committee-transcripts
Polish Sejm committee transcripts — full API coverage (terms 9 and 10)
Official committee transcripts ("pełny zapis przebiegu posiedzenia") from the Sejm
of the Republic of Poland, parsed into individually attributed speaker turns.
Scope
Committees: all standing committees with zapis PDFs in the Sejm API.
Terms: 9 (2019-11-14 → 2023-11-09) and 10 (2023-11-14 → 2026-09-17).
Provider and primary source: Kancelaria Sejmu RP, https://api.sejm.gov.pl/… See the full description on the dataset page: https://huggingface.co/datasets/PiotrSty/sejm-committee-transcripts.exp-pool-commit-code-raw
Locus EXP Commit Code - shuffled raw proxy pool
Deterministically shuffled commit-message and unified-diff documents with complete source metadata.
MANIFEST.json pins source identity, sampling policy, token budgets, and per-file checksums.
The paired Dolma-2-tokenized repository preserves prompt masking for reproducible proxy training.
so101_poker_play
so101_poker_play
This dataset was generated using a phospho starter pack.
This dataset contains a series of episodes recorded with a robot and multiple cameras. It can be directly used to train a policy using imitation learning. It's compatible with LeRobot and RLDS.
commitpack-ft-instructOctocode's CommitPackFT in Alpaca instruction format, with several randomly selected natural language preludes to the commit messages to make them better resemble a user request.
When the instruction, old code, and new code combined are small enough to fit within 4096 Llama tokens the output is usually the full contents of the file after a commit. Otherwise, the output will be a sequence of ndiff chunks with up to five lines of context each.
An example:
```ndiff
from… See the full description on the dataset page: https://huggingface.co/datasets/chargoddard/commitpack-ft-instruct.