Poindexter-Labs/PDL-SWE-Bench

PDL-SWE-Bench

PDL-SWE-Bench is an agentic software-engineering benchmark maintained by Poindexter Labs — the SWE sibling of PDL-Bench. Each task drops an agent into an original, internally-authored code repository with an engineering issue written as prose, a passing public test suite, and a fixed token budget. The agent's submitted patch is graded against a held-out acceptance suite it never saw during the episode.

Like PDL-Bench, this is an open benchmark (HLE-style): the complete answer key ships with the dataset — the reference (gold) patch, the held-out tests, and the test-level grading contract — so anyone can grade patches locally and reproduce results. See Contamination.

The row schema follows the SWE-bench conventions (problem_statement, patch, test_patch, FAIL_TO_PASS, PASS_TO_PASS) so existing tooling maps over directly, with one difference: these are self-contained synthetic repositories, shipped inline in the files column and as browsable trees under tasks/, rather than references to public GitHub commits. Nothing in this dataset derives from public repositories or issue trackers.

Difficulty tiers

18 tasks span five calibrated tiers. Tier placement is gated by measurement at authoring time, not intuition:

Certification claims are dated, pre-publication facts: they were measured before this dataset was released. Results obtained on these tasks by models trained after publication carry no such evidentiary weight.

Reward model

Episodes are graded on a continuous scale rather than binary resolution:

reward = 0.80 * hidden_pass_fraction + 0.15 * public_regression + 0.05 * build_health

• —1.0 — complete fix; 0.2 — the "do-nothing floor" (public tests still pass, build healthy); between — genuine partial fixes; below 0.2 — the patch broke working code.
• —Hidden suites are built as graded ladders: each task shipped with author-replayed partial solutions at known reward values, registered before any model ran (graded_ladder_rungs lists them). In measured campaigns, frontier models repeatedly landed exactly on these pre-registered values — a landed rung names the specific machinery a model failed to build.

Schema

Browsable copies live under tasks/<task_id>/: the task manifest.yaml, the fixture tree, the held-out tests (hidden/), and the reference patch (solution/gold.diff).

Evaluating a model

1. 1.Materialize files into a working directory (or copy tasks/<task_id>/fixtures/...).
2. 2.Give the agent problem_statement, the repository, and tools; enforce patch_policy_* and token_budget.
3. 3.Apply the agent's patch to a pristine copy; apply test_patch; run PASS_TO_PASS and FAIL_TO_PASS (pytest); compute the reward formula.

The numbers published by Poindexter Labs additionally run inside a network-isolated, container-sandboxed harness with full episode provenance (model route, prompt version, image digests, trajectories) and calibrated baselines executed in-campaign (reference patch 1.0, do-nothing agent 0.2, on every task). Evaluation-as-a-service against the same harness — including private, uncontaminated task sets — is available; contact Poindexter Labs.

Contamination

Publishing tasks and answer keys means future training corpora will likely absorb them; that trade-off is accepted deliberately, as with PDL-Bench. These tasks were contamination-clean at authoring (original codebases, original defects, nothing derived from public sources), and all published certification results predate this release. Poindexter Labs maintains unpublished task sets under the same methodology for measurement that must remain contamination-free.

License

CC-BY-4.0.