CoolFace
Datasetpublic

SEC-bench/SEC-bench-Pro

SEC-bench-Pro SEC-bench-Pro is a benchmark dataset of real-world security vulnerabilities in JavaScript engines (V8 and SpiderMonkey). Each instance contains a verified vulnerability with its Docker-reproducible environment, detailed description, and ground-truth fix patch. Dataset Summary Total instances: 183 V8 (Chromium): 103 instances SpiderMonkey (Firefox): 80 instances Vulnerability types: 24 distinct categories (type confusion, use-after-free, sandbox… See the full description on the dataset page: https://huggingface.co/datasets/SEC-bench/SEC-bench-Pro.

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face
1likes151downloads
Dataset Card

SEC-bench-Pro

SEC-bench-Pro is a benchmark dataset of real-world security vulnerabilities in JavaScript engines (V8 and SpiderMonkey). Each instance contains a verified vulnerability with its Docker-reproducible environment, detailed description, and ground-truth fix patch.

Dataset Summary

  • —Total instances: 183
  • —V8 (Chromium): 103 instances
  • —SpiderMonkey (Firefox): 80 instances
  • —Vulnerability types: 24 distinct categories (type confusion, use-after-free, sandbox bypass, OOB read/write, etc.)

Data Instances

Each row in the dataset represents a single vulnerability instance with the following fields:

FieldTypeDescription
instance_idstringUnique identifier in the format {project}__{bug_id} (e.g., v8__446124893, sm__1968423).
projectstringTarget project: v8, sm (SpiderMonkey), etc.
bug_idstringBug tracker ID (Chromium issue ID for V8, Bugzilla ID for SpiderMonkey).
image_namestringDocker image name for reproducing the vulnerable environment (e.g., hwiwonlee/v8.x86_64:446124893).
work_dirstringWorking directory inside the Docker container where the engine source is located (e.g., /src/v8, /src/gecko-dev).
verification_binarystringPath to the built binary used to trigger the vulnerability (e.g., out/x64.asan/d8, /out/js).
command_optionsstringCommand-line flags needed to trigger the vulnerability (e.g., --allow-natives-syntax --experimental-wasm-exnref). May be empty.
target_source_fileslist[string]Source files that contain the vulnerability and need to be patched.
target_subdirlist[string]Subdirectories within the engine source that are relevant to the vulnerability.
target_vulnerability_typestringClassification of the vulnerability (e.g., Type confusion, Use-after-free, Out-of-bounds write).
error_typestringHow the vulnerability manifests when triggered: ASAN_CRASH, RUNTIME_CRASH, DCHECK, SANDBOX_VIOLATION, etc.
descriptionstringDetailed technical description of the vulnerability, including root cause analysis and exploitation potential.
vrpstringVulnerability Reward Program (bug bounty) value in USD, if applicable. Empty string or "none" if not awarded.
fix_patcheslist[string]Full text of the patch file(s) that fix the vulnerability, in unified diff format.

Usage

python
from datasets import load_dataset

# Load all instances (default config)
ds = load_dataset("SEC-bench/SEC-bench-Pro", split="test")

# Load only V8 instances
v8_ds = load_dataset("SEC-bench/SEC-bench-Pro", "v8", split="test")

# Load only SpiderMonkey instances
sm_ds = load_dataset("SEC-bench/SEC-bench-Pro", "sm", split="test")

# Filter by project
v8_only = ds.filter(lambda x: x["project"] == "v8")

# Filter by vulnerability type
type_confusion = ds.filter(lambda x: x["target_vulnerability_type"] == "Type confusion")

Citation

If you use this dataset, please cite:

bibtex
@misc{lee2026secbenchprolanguagemodels,
      title={{SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?}}, 
      author={Hwiwon Lee and Jiawei Liu and Dongjun Kim and Ziqi Zhang and Chunqiu Steven Xia and Lingming Zhang},
      year={2026},
      eprint={2605.26548},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2605.26548}, 
}