DogukanUrker/PatchEval

PatchEval Pilot 20

PatchEval is a frozen benchmark for agentic coding. Each task starts from the parent of a real Python bug-fix commit whose regression test landed with the fix. The agent receives the reviewed GitHub issue and parent source; scoring is deterministic from hidden fail-to-pass and regression test exit codes.

This immutable pilot-20 release contains 20 tasks mined from recent commits and validated in four cells:

1. 1.the hidden regression test fails on the parent;
2. 2.the parent's previously passing regression suite stays green;
3. 3.the hidden regression test passes on the original fix; and
4. 4.the regression suite stays green after the original fix.

The release publishes no Docker images. BenchKit builds each task runtime locally from its frozen recipe and removes its task-specific Docker resources after the run.

Contents

• —dataset.json: release manifest and task IDs
• —tasks.jsonl: prompts, commands, runtime recipes, artifact paths, and hashes
• —sources/: deterministic parent-commit source archives without .git
• —hidden-tests/: grader patches; these remain outside the agent sandbox
• —attestations/: validation records produced before freezing
• —SHA256SUMS: checksums for every frozen release artifact
• —miner/: the source, configuration, schemas, tests, and documentation used to mine, validate, review, and freeze the release

Run with BenchKit

Download an immutable snapshot, point BenchKit at its root, and run the stock Pi harness:

export BENCHKIT_PATCHEVAL_DATASET=/path/to/PatchEval
uv run benchkit --headless --models MODEL --benchmarks patcheval \
  --harness pi --repair-attempts 1

The agent sandbox contains only the checksummed parent source and a restricted inference route. It has no Git history, network egress, hidden tests, gold patch, host mount, or Docker socket. Hidden grader output is never returned to the model; a repair attempt receives only generic feedback.

Integrity

Verify the release after download:

sha256sum --check SHA256SUMS

dataset.json additionally pins the SHA-256 of tasks.jsonl. A frozen release is never regenerated or changed in place; a later mining window becomes a new version.

Reproduce the miner

The miner is intentionally separate from BenchKit. From miner/:

uv sync
uv run patcheval-mine --help
uv run pytest

Discovery, automated validation, blind review, and freezing are separate fail-closed gates. See miner/README.md and miner/docs/review-policy.md for the complete method and publication boundary.

Licensing

The miner package declares Apache-2.0 in miner/pyproject.toml. Source snapshots, issue text, and regression-test patches originate from the named upstream repositories and remain subject to their respective licenses. Consult each task's repository field and upstream project before redistribution or reuse.