g-for-gour/llm-commit-message-evaluation

Dataset Card for LLM Commit Message Evaluation

The LLM Commit Message Evaluation dataset is designed to evaluate and compare the performance of Large Language Models (LLMs) in generating high-quality git commit messages. It contains real-world code diffs, issue descriptions, and issue titles extracted from open-source repositories (such as OWASP/Nest).

For each code change, the dataset provides the original human-written commit message alongside commit messages generated by three state-of-the-art LLMs: ChatGPT, Claude, and Gemini. Crucially, the dataset includes comprehensive evaluation metrics:

• —Human Evaluation: Scored by multiple annotators on Code Coverage, Issue Coverage, and Commit Readability.
• —Automated Evaluation: Scored using BLEU, BERTScore (F1), and CodeBERT Similarity.

Dataset Details

Dataset Description

• —Curated by: Gour Gupal Talukder and Amit Kumar Sharma
• —Shared by: Gour Gupal Talukder
• —Language(s) (NLP): English (en), alongside programming languages (Python, Makefile, TypeScript, etc.) represented in code diffs
• —License: MIT

Dataset Sources

• —Repository: https://huggingface.co/datasets/gourgupal/llm-commit-message-evaluation
• —Paper: [More Information Needed]
• —Demo: [More Information Needed]

Uses

Direct Use

• —`text-generation`: Generating commit messages from code diffs and issue context.
• —`text-evaluation`: Training and evaluating models or reward systems on the quality of generated code documentation and commit messages.
• —Benchmarking: Comparing LLMs on software engineering-specific text generation tasks.

Out-of-Scope Use

• —Automated deployment systems that apply commit messages without human oversight in production environments.
• —General-purpose NLP evaluations unrelated to software engineering or code context.

Dataset Structure

Data Fields

Core Context Fields

• —id / _id: Unique identifiers for the data instance.
• —codeDiff: The raw git diff representing the code changes.
• —codeDiffSummary: A high-level summary of the code diff.
• —issueTitle: The title of the associated GitHub/GitLab issue.
• —issueDescription: The detailed description of the associated issue.
• —commitLink: URL to the original commit in the source repository.
• —issueNo: The issue number from the source repository.

Commit Messages

• —Original: The actual commit message written by the human developer.
• —ChatGPT Generated: Commit message generated by ChatGPT.
• —Claude Generated: Commit message generated by Claude.
• —Gemini Generated: Commit message generated by Gemini.

Human Evaluation Metrics

(Evaluated by multiple human annotators, denoted by `ev1`, `ev2`, etc.)

• —ev1_evaluatorName / ev2_evaluatorName: Names of the human annotators (e.g., Shakirul, Anindo Sarker).
• —codeCoverage_commit[X]_ev[Y]: Score indicating how well the commit message covers the code changes.
• —issueCoverage_commit[X]_ev[Y]: Score indicating how well the commit message addresses the issue context.
• —commitReadability_commit[X]_ev[Y]: Score indicating the readability and clarity of the commit message.
• —finalMarks_commit[X]_ev[Y]: The aggregated final score given by the evaluator.

Automated Evaluation Metrics

(Calculated for Original, ChatGPT, Claude, and Gemini)

• —bertscore_f1_[Model]: BERTScore F1 metric evaluating semantic similarity.
• —bleu_score_[Model]: BLEU score evaluating n-gram overlap.
• —codebert_similarity_[Model]: CodeBERT similarity score, highly relevant for software engineering text.

Dataset Creation

Curation Rationale

Writing clear, descriptive commit messages is a time-consuming task for developers. While LLMs show promise in automating this, there is a need for high-quality datasets that not only provide generated text but also rigorously evaluate it using both human judgment and domain-specific automated metrics (like CodeBERT).

Source Data

Data Collection and Processing

Data is sourced from open-source GitHub repositories (such as OWASP/Nest). Commits were paired with their corresponding issues to provide full context (why the change was made + what the change is).

Who are the source data producers?

Open-source developers contributing to public repositories on GitHub.

Annotations

Annotation process

Human evaluators manually reviewed the code diffs and issue descriptions, then blindly scored the commit messages (Original + LLM generated) on a defined scale for Readability, Code Coverage, and Issue Coverage.

Who are the annotators?

Software developers and researchers familiar with git workflows and code review processes (e.g., Shakirul, Anindo Sarker).

Personal and Sensitive Information

The dataset contains public code diffs and public issue discussions from open-source projects. Personal identifiers are limited to publicly visible evaluator names and open-source commit metadata.

Bias, Risks, and Limitations

• —The dataset currently focuses on specific open-source repositories. Coding styles, issue description formats, and commit standards may reflect those communities and might not fully represent proprietary or enterprise codebases.
• —Human evaluations carry inherent subjectivity, though multiple evaluators were used to mitigate individual bias.

Recommendations

Users should keep in mind that performance metrics on this dataset evaluate commit generation in an open-source setting; performance may vary when transferred to private, domain-specific, or multi-language codebases.

Citation

BibTeX:

@misc{talukder2026llmcommiteval,
  author = {Gour Gupal},
  title = {LLM Commit Message Evaluation Dataset},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{[https://huggingface.co/datasets/g-for-gour/llm-commit-message-evaluation](https://huggingface.co/datasets/g-for-gour/llm-commit-message-evaluation)}}
}