project-themis/git-commits
Themis-Git-Commits Overview Themis-Git-Commits is a large-scale dataset of single-file code commits mined from permissively licensed GitHub repositories via the BigQuery GitHub public dataset. The SQL query restricts to repositories under permissive open-source licenses only (MIT, Apache-2.0, BSD-2/3-Clause, ISC, CC0-1.0, EPL-1.0, MPL-2.0, Unlicense, AGPL-3.0, LGPL-2.1, Artistic-2.0). The BigQuery snapshot used contains commits up to early 2022 —… See the full description on the dataset page: https://huggingface.co/datasets/project-themis/git-commits.
13k
1---2tags:3- code4- github5- commits6- multilingual7license: apache-2.08task_categories:9- text-generation10language:11- code12size_categories:13- 10M<n<100M14---15 16 17<div align="center">18 19# Themis-Git-Commits20 21[](https://arxiv.org/abs/2605.00754)22[](https://huggingface.co/collections/project-themis/themis-reward-model-collection)23[](https://huggingface.co/collections/project-themis/themis-preference-datasets-and-benchmarks)24[](https://github.com/iNeil77/Themis)25[](https://hub.docker.com/repository/docker/ineil77/themis/general)26 27</div>28 29## Overview30 31**Themis-Git-Commits** is a large-scale dataset of single-file code commits mined from **permissively licensed** GitHub repositories via the [BigQuery GitHub public dataset](https://console.cloud.google.com/marketplace/product/github/github-repos). The SQL query restricts to repositories under permissive open-source licenses only (MIT, Apache-2.0, BSD-2/3-Clause, ISC, CC0-1.0, EPL-1.0, MPL-2.0, Unlicense, AGPL-3.0, LGPL-2.1, Artistic-2.0). The BigQuery snapshot used contains commits up to **early 2022** — predating the widespread availability of LLM code generation tools — ensuring that all code changes in the dataset represent **genuine human-authored preferences**.32 33This is the **raw commit dataset** — prior to merging with pull request data to subset only for merged commits. It serves as the foundational data source for the commit-based preference pairs in [Themis-CodePreference](https://huggingface.co/datasets/project-themis/Themis-CodePreference), which is used to train the [Themis-RM](https://huggingface.co/collections/project-themis/themis-reward-model-collection) suite of multilingual code reward models.34 35Each row represents a single commit that changes exactly one file in a repository with a permissive open-source license. The dataset includes the commit metadata (SHA, message, timestamp, license) along with the pre-commit and post-commit file contents, enabling downstream construction of code-change preference pairs across multiple quality dimensions.36 37## Collection Pipeline38 39The commit mining pipeline is described in detail in the [Themis paper](https://arxiv.org/abs/2605.00754) and the [Dataset](https://github.com/iNeil77/Themis/tree/main/Dataset) folder in the GitHub repository. The BigQuery SQL query and scraping infrastructure are modified from the [OctoPack](https://arxiv.org/abs/2308.07124) pipeline ([CommitPack](https://huggingface.co/datasets/bigcode/commitpack)); the subsequent filtering, classification, and preference construction stages are original to Themis. At a high level:40 411. **BigQuery Mining** — A [GoogleSQL query](https://github.com/iNeil77/Themis/blob/main/Dataset/Commit_Mining_SQL/consolidated_query.sql) (modified from [OctoPack](https://arxiv.org/abs/2308.07124)) extracts single-file commits from `bigquery-public-data.github_repos`, filtering for permissive licenses, target programming languages, and non-trivial commit messages.42 432. **Repository Reputation Filtering** — Commits are subset to those originating from [curated high-reputation repositories](https://github.com/iNeil77/Themis/tree/main/Dataset/Repos) (15+ GitHub stars, 5+ contributors, 10+ issues).44 453. **Content Retrieval** — The pre-commit (`old_contents`) and post-commit (`new_contents`) file contents are fetched from GitHub via shallow git fetches using [retrieve_commit_contents.py](https://github.com/iNeil77/Themis/blob/main/Dataset/Utils/retrieve_commit_contents.py).46 474. **MinHash Deduplication** — Near-duplicate content is removed using [MinHash LSH deduplication](https://github.com/iNeil77/Themis/blob/main/Dataset/Utils/minHash_dedupe_local.py) (shingle size 5, 256 permutations, Jaccard threshold 0.7).48 49## Downstream Processing (Not in This Dataset)50 51The steps below are applied downstream and are **not** reflected in this raw dataset:52 53- **Extension Filtering** — Commits are filtered so the changed file's extension matches a target programming language. Applied in [Themis-Git-Commits-Merged](https://huggingface.co/datasets/project-themis/git-commits-merged).54- **Pull Request Cross-Referencing** — Commits are cross-referenced with [GHTorrent](https://ghtorrent.org/) pull request data (through end of 2021) to retain only non-reverted commits that are part of successfully merged pull requests, ensuring implicit human validation. Applied in [Themis-Git-Commits-Merged](https://huggingface.co/datasets/project-themis/git-commits-merged).55- **Temporal Subsetting** — For training data ([Themis-CodePreference](https://huggingface.co/datasets/project-themis/Themis-CodePreference)), only commits pushed before **March 2019** are retained. For benchmark data ([Themis-CodeRewardBench](https://huggingface.co/datasets/project-themis/Themis-CodeRewardBench)), commits are scoped to **June 2019 – January 2021** from disjoint repositories.56- **Aspect Classification** — Commits are assigned to quality dimensions (Functional Correctness, Runtime Efficiency, Memory Efficiency, Security Hardness, Readability & Maintainability) using criteria-specialized [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) commit classifiers, trained on seed positives retrieved via [curated term lists](https://github.com/iNeil77/Themis/tree/main/Dataset/Commit_Mining_Terms).57- **LLM Scoring & Instruction Synthesis** — Frontier LMs validate preference strength and generate realistic inverse instructions.58- **LLM-as-a-Judge Preference Labelling** — Three frontier LMs (DeepSeek-V3.2-Speciale, Kimi-K2.5, MiniMax-M2.5) each label preferences via multi-sample majority voting; only labels on which all three unanimously agree are retained.59 60## Dataset Schema61 62<div align="center">63 64| Column | Type | Description |65|:---|:---:|:---|66| `commit` | string | Git commit SHA |67| `subject` | string | First line of the commit message |68| `message` | string | Full commit message body |69| `repos` | string | Comma-separated list of repository names containing this commit |70| `file_path` | string | Path of the changed file |71| `license` | string | SPDX license identifier of the source repository |72| `unix_time` | int64 | Committer timestamp (seconds since epoch) |73| `new_contents` | string | File contents after the commit (post-commit) |74| `old_contents` | string | File contents before the commit (pre-commit) |75 76</div>77 78## Filters Applied During Mining79 80<div align="center">81 82| Filter | Purpose |83|:---|:---|84| **License allowlist** | MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, CC0-1.0, EPL-1.0, MPL-2.0, Unlicense, AGPL-3.0, LGPL-2.1, Artistic-2.0 |85| **Language allowlist** | Python, Java, JavaScript, C, C#, C++, TypeScript, Go, Ruby |86| **Message length** | 10 < length < 15,000 characters |87| **Message blocklist** | ~50 low-signal messages excluded (e.g., "initial commit", "wip", "yolo") |88| **Pattern exclusion** | Merge commits and CI push messages filtered out |89| **Same-path constraint** | `old_path = new_path` — file was modified in place, not renamed or moved |90| **Single-file constraint** | Commit touches exactly one file |91| **Content retrieval** | Both pre-commit and post-commit file contents successfully fetched |92| **Near-deduplication** | MinHash LSH with Jaccard threshold 0.7 |93 94</div>95 96## Usage97 98```python99from datasets import load_dataset100 101dataset = load_dataset("project-themis/git-commits")102 103# Inspect a sample104sample = dataset["train"][0]105print(f"Commit: {sample['commit']}")106print(f"Subject: {sample['subject']}")107print(f"License: {sample['license']}")108print(f"File: {sample['file_path']}")109print(f"Old contents length: {len(sample['old_contents'])}")110print(f"New contents length: {len(sample['new_contents'])}")111```112 113## License114 115This dataset is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). The source commits are drawn exclusively from repositories with permissive open-source licenses (see filter table above).116 117## Citation118 119```bibtex120@article{themis2025,121 title={Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring},122 author={Paul, Indraneil and Gurevych, Iryna and Glava\v{s}, Goran},123 journal={arXiv preprint arXiv:2605.00754},124 year={2025}125}126```