Datasetpublic

ashishkosana/review-lens-evals

Review Lens Evals A small, hand-labeled evaluation set for measuring the precision and recall of LLM code-review systems on unified diffs. The dataset accompanies Ashishkosana/review-lens, a multi-lens reviewer that examines correctness, security, performance, and test coverage before running a separate adversarial verification pass. Why this dataset exists Code-review evaluations need both positive and negative cases. The four seeded bug diffs test whether a… See the full description on the dataset page: https://huggingface.co/datasets/ashishkosana/review-lens-evals.

sourceHugging Facemitupdated 9d agoView on Hugging Face

0likes45downloads

Dataset Card

Review Lens Evals

A small, hand-labeled evaluation set for measuring the precision and recall of LLM code-review systems on unified diffs.

The dataset accompanies `Ashishkosana/review-lens`, a multi-lens reviewer that examines correctness, security, performance, and test coverage before running a separate adversarial verification pass.

Why this dataset exists

Code-review evaluations need both positive and negative cases. The four seeded bug diffs test whether a reviewer can locate a known issue. The two clean diffs test whether it can avoid inventing findings. A warning on a clean diff is a false positive and reduces precision.

This is a transparent seed set, not a claim of broad benchmark coverage.

Case	Expected outcome
`sqli`	Blocker security finding for string-concatenated SQL
`off_by_one`	High-severity correctness finding
`n_plus_one`	High-severity performance finding
`missing_test`	Medium-severity test-coverage finding
`clean_rename`	No finding
`clean_guard`	No finding

Schema

—case_id: stable case identifier
—description: human-written ground truth
—diff: complete unified diff presented to the reviewer
—labels: expected findings with file, line, lens, and severity
—is_clean: whether no finding is expected
—source_url: corresponding source file in the project repository

Intended use

Use this dataset to:

1.Run a code-review system on each diff.
2.Match predictions to labels by file and lens, with a documented line tolerance.
3.Count unmatched predictions as false positives.
4.Report precision, recall, and F1 across the complete corpus.

The reference implementation uses one-to-one matching so duplicate comments on the same defect do not inflate recall.

Limitations

—Six synthetic Python diffs are not representative of all languages, repositories, or vulnerability classes.
—Labels identify the intentionally seeded issue; they do not prove that no other interpretation is possible.
—Results should not be used as evidence that a reviewer is safe to merge code automatically.
—LLM reviewers remain vulnerable to prompt injection in attacker-authored source and diff text.

Source and license