Anonymous648/zkml-audit-benchmark

zkml-audit-benchmark

A benchmark dataset for evaluating AI agents on zkML soundness auditing: finding cryptographic vulnerabilities in zero-knowledge machine learning proof implementations.

Overview

This dataset pairs 4 published zkML research papers with their corresponding frozen codebase snapshots and 56 bug artifacts (20 real-world from expert audits + 36 synthetic for broader coverage). Each artifact describes a single soundness vulnerability — the code edits to inject it, ground-truth labels for scoring, and presence probes for post-injection validation

The benchmark supports two complementary workflows:

1. 1.Pair extraction — load a (paper, codebase) pair for an agent to audit.
2. 2.Test-case generation — sample artifacts and apply their edit logic to the clean codebase, producing flawed codebases with known ground-truth findings.

Documentation

• —DATASHEET.md — datasheet for this dataset, following Gebru et al. 2021.
• —CONTRIBUTING.md — how to add new (paper, codebase) pairs and bug artifacts.
• —CHANGELOG.md — release history and label/codebase corrections.
• —schema/artifact.v2.schema.json — authoritative JSON schema for bug artifacts.

Positioning vs. Existing Security Benchmarks

Several mature benchmarks evaluate code-security tooling, but none target the theory-to-implementation gap that defines zkML soundness. The table below contrasts the design axes that matter for this setting: domain coverage, the granularity of the ground-truth labels, and whether artifacts are paired with the academic claims they implement.

Juliet ships tens of thousands of injected vulnerabilities for general-purpose code, but its labels are generic CWE classes rather than per-paper soundness claims. The OWASP Benchmark is web-application focused, and the NIST Cryptographic Algorithm Validation Program validates implementations of standardized primitives, not bespoke proof-system protocols. By contrast, this benchmark ships paper–codebase pairs together with declarative artifact specifications that re-introduce a precisely characterized soundness gap, making it directly suitable for studying theory-to-practice alignment in zkML.

Dataset Structure

Configs

Loading

from datasets import load_dataset

pairs = load_dataset("anonymous-zkml-benchmark/zkml-audit-benchmark", "pairs")
artifacts = load_dataset("anonymous-zkml-benchmark/zkml-audit-benchmark", "artifacts")

Raw Files

Beyond the Parquet tables, the repository includes:

• —papers/{pair_id}.pdf — research paper PDFs
• —codebases/{pair_id}.zip — frozen codebase snapshots (Git LFS)
• —artifacts/{pair_id}/*.json — full artifact JSONs with edit instructions, conflict metadata, and presence probes
• —schema/artifact.v2.schema.json — JSON Schema defining the artifact format

The Parquet tables contain flattened metadata for filtering and loading. The full artifact JSONs (with heterogeneous edit/probe structures) are the authoritative source for test-case generation.

Data Fields

pairs config

artifacts config

Pair Inventory

Artifact Summary

• —Total artifacts: 56 (14 zkGPT + 13 zkLLM + 14 zkML + 15 zkTorch)
• —Source breakdown: 20 real (derived from expert audit reports), 36 synthetic (authored for broader coverage)
• —Each artifact includes declarative code edits, conflict metadata for safe composition, and presence probes for post-injection validation

Reproducibility

All files are checksummed in MANIFEST.json. To verify integrity:

python scripts/verify_dataset.py

To rebuild the Parquet tables from the in-repo artifact JSONs:

python scripts/build_parquet.py

Known Limitations & Assumptions

1. 1.Paper ↔ codebase mapping uses fixed pair IDs (zkllm, zkml, zktorch, zkgpt) defined in scripts/build_parquet.py.

1. 1.Non-git snapshots: The zktorch codebase is a directory snapshot dated 2026-04-19 and cannot be pinned to an upstream commit. zkllm and zkml are pinned to Git commits (993311ea… and 4378958… respectively). zkgpt is pinned to Zenodo record 14727819 v1.

1. 1.Paper licensing: PDFs are included for research reproducibility. The dataset-level CC-BY-4.0 license covers only the curation layer (artifact definitions, schema, scripts). Papers carry their respective publisher terms (ACM, arXiv). Users redistribute at their own responsibility.

1. 1.Artifact format: Artifacts follow schema/artifact.v2.schema.json. All graded labels live under finding.labels.

1. 1.Scope: This release covers 4 of the 10 available research papers — specifically those with paired frozen codebases and authored artifacts. Remaining papers may be added in future releases.

Schema Reference

The artifact JSON format is defined by schema/artifact.v2.schema.json. Key structures:

• —finding.labels — the two graded fields (relevant_code, paper_reference)
• —edits — ordered list of code edits to inject the bug
• —conflict_keys — files, regions, and semantic tags for safe composition
• —presence_probes — assertions to verify successful injection

Citation

If you use this dataset, please cite:

@misc{zkml-audit-benchmark,
  title={zkml-audit-benchmark: A Benchmark for AI Agents on zkML Soundness Auditing},
  author={Anonymous},
  year={2026},
  url={https://huggingface.co/datasets/anonymous-zkml-benchmark/zkml-audit-benchmark},
}

License

• —Dataset curation layer (artifacts, schema, scripts, documentation): CC-BY-4.0
• —Codebases: retain their original upstream licenses (see each codebase's LICENSE file inside the zip)
• —Papers: subject to respective publisher terms