project-themis/git-commits-merged

<div align="center">

Themis-Git-Commits-Merged

    

</div>

Overview

Themis-Git-Commits-Merged is a large-scale dataset of ~3.98M single-file code commits from permissively licensed GitHub repositories that have been cross-referenced with GHTorrent pull request data to retain only commits that are part of successfully merged, non-reverted pull requests. This provides implicit human validation of each code change — a merge decision by project maintainers confirms the intent and quality of the commit.

This dataset is derived from Themis-Git-Commits (the raw, unfiltered commit pool) by joining against GHTorrent pull request records through end of 2021 (the temporal extent of the GHTorrent PR data used). It serves as the direct input to the aspect classification and preference construction stages of the Themis pipeline, which produces the commit-based preference pairs in Themis-CodePreference and Themis-CodeRewardBench.

The SQL query used for initial commit mining restricts to repositories under permissive open-source licenses only (MIT, Apache-2.0, BSD-2/3-Clause, ISC, CC0-1.0, EPL-1.0, MPL-2.0, Unlicense, AGPL-3.0, LGPL-2.1, Artistic-2.0). The BigQuery snapshot used contains commits up to early 2022 — predating the widespread availability of LLM code generation tools — ensuring that all code changes in the dataset represent genuine human-authored preferences.

Relationship to Themis-Git-Commits

<div align="center">

</div>

Collection Pipeline

The commit mining pipeline is described in detail in the Themis paper and the Dataset folder in the GitHub repository. The BigQuery SQL query and scraping infrastructure are modified from the OctoPack pipeline (CommitPack); the subsequent filtering, classification, and preference construction stages are original to Themis.

1. 1.BigQuery Mining — A GoogleSQL query (modified from OctoPack) extracts single-file commits from permissively licensed repositories in bigquery-public-data.github_repos, filtering for target programming languages and non-trivial commit messages.

1. 1.Repository Reputation Filtering — Commits are subset to those originating from curated high-reputation repositories (15+ GitHub stars, 5+ contributors, 10+ issues).

1. 1.Extension Filtering — Commits are further filtered so the changed file's extension matches a target programming language.

1. 1.Content Retrieval — The pre-commit (old_contents) and post-commit (new_contents) file contents are fetched from GitHub via shallow git fetches using retrieve_commit_contents.py.

1. 1.MinHash Deduplication — Near-duplicate content is removed using MinHash LSH deduplication (shingle size 5, 256 permutations, Jaccard threshold 0.7).

1. 1.Pull Request Cross-Referencing (this step produces this dataset) — Commits are joined with GHTorrent pull request data (through end of 2021) to retain only non-reverted commits that are part of successfully merged pull requests from reputable repositories, ensuring implicit human validation of each code change.

Downstream Processing (Not in This Dataset)

The steps below are applied downstream to produce the final preference pairs in Themis-CodePreference and Themis-CodeRewardBench, and are not reflected in this dataset:

• —Language Subsetting — The 24 languages in this dataset are narrowed to the 8 target languages (C, C#, C++, Go, Java, JavaScript, Python, Ruby) used in Themis-CodePreference and Themis-CodeRewardBench.
• —Temporal Subsetting — For training data (Themis-CodePreference), only commits pushed before March 2019 are retained. For benchmark data (Themis-CodeRewardBench), commits are scoped to June 2019 – January 2021 from disjoint repositories.
• —Aspect Classification — Commits are assigned to quality dimensions (Functional Correctness, Runtime Efficiency, Memory Efficiency, Security Hardness, Readability & Maintainability) using criteria-specialized ModernBERT commit classifiers, trained on seed positives retrieved via curated term lists.
• —LLM Scoring & Instruction Synthesis — Frontier LMs validate preference strength and generate realistic inverse instructions.
• —LLM-as-a-Judge Preference Labelling — Three frontier LMs (DeepSeek-V3.2-Speciale, Kimi-K2.5, MiniMax-M2.5) each label preferences via multi-sample majority voting; only labels on which all three unanimously agree are retained.

Dataset Schema

<div align="center">

</div>

Language Distribution

The dataset is partitioned by programming language, with one config per language. Each config has a single train split.

<div align="center">

</div>

Filters Applied

All filters from the upstream Themis-Git-Commits pipeline apply, plus the PR merge filter:

<div align="center">

</div>

Usage

from datasets import load_dataset

# Load a single language
python_commits = load_dataset("project-themis/git-commits-merged", "Python")
sample = python_commits["train"][0]
print(f"Commit: {sample['commit']}")
print(f"Subject: {sample['subject']}")
print(f"License: {sample['license']}")
print(f"File: {sample['file_path']}")

# Load all languages
for lang in ["C", "CSharp", "Cpp", "Go", "Java", "JavaScript", "Python", "Ruby", "TypeScript"]:
    ds = load_dataset("project-themis/git-commits-merged", lang)
    print(f"{lang}: {len(ds['train'])} commits")

License

This dataset is released under the Apache 2.0 License. The source commits are drawn exclusively from repositories with permissive open-source licenses (see filter table above).

Citation

@article{themis2025,
  title={Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring},
  author={Paul, Indraneil and Gurevych, Iryna and Glava\v{s}, Goran},
  journal={arXiv preprint arXiv:2605.00754},
  year={2025}
}