ai-sec-lab/PatchBench

PatchBench

  

PatchBench is a benchmark for evaluating AI agents on realistic vulnerability patching tasks: 213 tasks drawn from 32 popular GitHub C/C++ projects. It selects vulnerabilities whose ground-truth fixes lie outside the crash stack, and uses vulnerability transplant plus code mutation to mitigate surface-level fixes and patch memorization.

This repository holds the task metadata, one row per task to identify the project, the exact repository state, the crash, and the reference patch.

Dataset Structure

Each example contains:

• —id: Unique task identifier, also naming the task's image (Same as the one in ARVO)
• —image: Docker Hub image for the task, b4drequest/vulpatch:{id}-vul
• —project: Real-world repository name
• —repo_addr: Repository URL
• —base_commit: Historical commit at which the vulnerability is still present, upstream of the developer's fixing commit
• —inverted_commit: Newer upstream commit the vulnerability was transplanted onto
• —fuzzer: Fuzzing engine that produced the crash, one of libfuzzer, honggfuzz, afl, aflpp
• —sanitizer_type: The error detection tool type
• —CWE_ID: The weakness the task's vulnerability manifests, derived from the sanitizer report and confirmed against the ground-truth patch
• —secondary_CWE_ID: The underlying defect behind it where the evidence pins one down, and None where the primary already states the weakness
• —command: Command that reproduces the crash inside the image, with the PoC at /tmp/poc
• —sanitizer_report: Sanitizer output for the original PoC, including the crash stack
• —patch: Reference patch against the task repository

base_commit, inverted_commit and patch are ground truth and are not visible to the agent during inference.

Dataset Usage

from datasets import load_dataset

dataset = load_dataset("ai-sec-lab/PatchBench", split="test")

Filtering works the usual way:

libxml2 = dataset.filter(lambda r: r["project"] == "libxml2")
ubsan   = dataset.filter(lambda r: r["sanitizer_type"] == "ubsan")

Evaluation

See our repo for details: ai-sec-lab/PatchBench.

Citation

@misc{shen2026PatchBench,
  title = {{{PatchBench}}: Evaluating {{AI}} Agents for Vulnerability Patching},
  author = {Shen, Chihao and Li, Jiacheng and Mahajan, Aastha and Tian, Jeffery Siyuan and Kwon, Yonghwi and Chen, Yizheng},
  year = 2026,
  number = {arXiv:2609.04075},
  eprint = {2609.04075}
}