cactus183/patchbench-dev

PatchBench

An OpenEnv environment where AI agents fix real bugs in real Python code — and are graded by actually running `pytest`.

PatchBench is a lightweight, reproducible benchmark for code-fixing agents. Unlike rule-based or string-matching graders, PatchBench writes each proposed patch to a sandboxed temp directory and runs pytest against it, scoring the agent on test pass-rate improvement while penalizing regressions — mirroring how a real CI system evaluates a pull request.

Why PatchBench

Existing code-agent benchmarks either test trivial toy problems or require multi-gigabyte container environments like SWE-bench. PatchBench targets the middle ground: genuine bug-fixing capability in scenarios that fit in a 2 vCPU / 8 GB container and run the full inference script in under 20 minutes.

The tasks span three difficulty tiers:

• —Easy — single-function, single-bug, obvious failure
• —Medium — multi-function files where the symptom appears downstream of the cause
• —Hard — files with multiple interacting bugs where the naive fix regresses other tests

Environment Interface

PatchBench implements the OpenEnv reset() / step() / state contract.

Observation

Action

Step Execution Flow

PatchBench follows the standard OpenEnv contract. Here is how a single step flows through the system:

+---------------+
|     Agent     |  (LLM via OpenAI client in inference.py)
+-------+-------+
        | PatchBenchAction(patched_code="...")
        v
+-----------------------------------------------+
|         PatchBenchEnv.step(action)            |
|                                               |
|  1. Increment step counter                    |
|  2. Call grader.grade_patch(...)              |
|  3. Update self.current_code if valid         |
|  4. Compute done flag                         |
+-------+---------------------------------------+
        |
        v
+-----------------------------------------------+
|              grader.grade_patch               |
|                                               |
|  1. ast.parse() -> syntax check               |
|  2. Write patch to tempfile                   |
|  3. subprocess.run(pytest, timeout=15)        |
|  4. Parse PASSED/FAILED/ERROR from stdout     |
|  5. Compute newly_passing, regressions        |
|  6. Apply reward shaping formula              |
|  7. Return (reward, info_with_breakdown)      |
+-------+---------------------------------------+
        | (reward, info)
        v
+-----------------------------------------------+
|       PatchBenchObservation returned          |
|                                               |
|  - buggy_code (updated if valid patch)        |
|  - failing_tests (human-readable summary)     |
|  - reward: [0.0, 1.0]                         |
|  - done: bool                                 |
|  - info: { reward_breakdown, test counts,     |
|            regressions, task metadata }       |
+-----------------------------------------------+

The subprocess sandbox uses tempfile.TemporaryDirectory() (auto-cleanup on context exit), a 15-second hard timeout, and inherits PATH and PYTHONPATH from the parent environment so pytest is importable across installation layouts. The grader never raises to the caller -- timeouts, crashes, and parse failures all return a valid reward.

Reward Function

PatchBench uses dense, shaped rewards that provide signal throughout the episode rather than only at termination.

Per step, the raw reward is computed as:

raw  = +0.3  if the patch is syntactically valid Python
     + 0.4 x (newly_passing_tests / originally_failing_tests)
     - 0.5 x number_of_regressions
     - 0.1  step cost
     + 0.3  if all tests now pass

The raw reward is then normalized into [0.0, 1.0] via (raw + 0.6) / 1.6 and clamped. This produces the following signal:

• —Invalid Python: near 0
• —Valid Python but no progress: small positive baseline
• —Partial progress: intermediate reward proportional to newly-passing tests
• —Regressions: strong penalty, can wipe out partial progress
• —Full success: terminal bonus pushes reward near 1.0

Every step's info dict includes a reward_breakdown field exposing each component's contribution (syntax bonus, progress reward, regression penalty, step cost, terminal bonus, raw sum, normalized value). This transparency lets researchers debug agent behavior and validate reward shaping decisions without reading the grader source.

Sample Episode Walkthrough

Here is an abbreviated trajectory of a baseline agent solving medium_01 (CSV parser with quoted-comma handling bug):

Reset -- Initial observation:

task_description: "Fix the CSV parser so it correctly handles quoted fields containing commas."

buggy_code:
    def _split_quoted(line):
        return line.split(",")

    def parse_row(line):
        return [field.strip() for field in _split_quoted(line)]

failing_tests:
    FAILING TESTS:
    test_row_with_multiple_quoted_commas
    test_row_with_quoted_comma

step_number: 0
max_steps: 8

Step 1 -- Agent submits a naive fix:

The agent tries to handle quotes with a regex replacement. Patch is valid Python but doesn't fully solve the problem.

reward: 0.44
info.reward_breakdown:
  syntax_valid_bonus: 0.3
  progress_reward: 0.2
  regression_penalty: 0.0
  step_cost: -0.1
  terminal_bonus: 0.0
  raw_sum: 0.4
  normalized: 0.44

Partial progress: one of the two failing tests now passes. The other still fails because the fix doesn't handle nested quoted commas.

Step 5 -- Agent converges on correct fix:

After 4 iterations of reading the updated failing_tests output and refining, the agent implements a proper state-machine parser that tracks quote state.

reward: 0.94
info.reward_breakdown:
  syntax_valid_bonus: 0.3
  progress_reward: 0.4
  regression_penalty: 0.0
  step_cost: -0.1
  terminal_bonus: 0.3
  raw_sum: 0.9
  normalized: 0.94

done: true
info.all_tests_pass: true

Why this matters for RL training: the 4-step plateau between step 1 and step 5 is exactly the kind of partial progress trajectory that dense reward functions are designed to exploit. A terminal-only reward would give the agent zero signal across steps 1-4, while PatchBench's shaped reward tells the agent "you're partially right, keep going." This is why the environment is well-suited for agent fine-tuning rather than just evaluation.

Tasks

PatchBench ships with 9 hand-crafted, pytest-verified tasks. Difficulty tiers describe structural complexity of the bug (single-file vs multi-function vs interacting), not model-vs-task win rates -- frontier models may one-shot structurally complex bugs when the full code is in context.

Each task directory contains buggy_code.py, test_code.py, and task.json with pre-computed baseline pass/fail sets.

Quickstart

Local

pip install -e .
python inference.py

Docker

docker build -t patchbench .
docker run -p 7860:7860 patchbench

The HTTP server exposes:

• —GET /health
• —GET / -- environment info and task list
• —POST /reset -- body {"seed": int?, "task_id": str?}
• —POST /step -- body {"action": {"patched_code": "..."}}
• —GET /state

Environment Variables

Baseline Scores

We ran the baseline inference script against Qwen/Qwen2.5-72B-Instruct via the HuggingFace Router on all 9 tasks.

Mean score across all tasks: 0.86

Full run logs: baseline_logs_full.txt

Observations

Qwen 2.5 72B one-shots 7 of 9 tasks (score 0.94 -- the one-step ceiling imposed by the step cost and terminal bonus formula). This is expected: with full file content and failing test names in context, frontier models have strong single-file debugging capability and do not need multi-step reasoning for localized bugs.

The signal of the environment emerges in two places:

1. 1.medium_01 (CSV quoted-comma parser): Qwen required all 8 steps with a 7-step plateau at 0.44 before converging at 0.94. This is the canonical dense-reward trajectory: partial progress, stuck state, hypothesis revision, convergence. Terminal-only reward functions would give zero signal across steps 1-7; PatchBench's shaped reward provides training signal at every step.

1. 1.hard_03 (MiniJSON encoder/decoder symmetry): Qwen made a partial fix on step 1 (score 0.35) that patched the encoder but broke the round-trip, then converged on step 2 (score 0.94) after seeing the new failing-tests output. This confirms the environment rewards reading the feedback loop, not just the initial guess.

What this tells us about benchmark design

The 0.94 ceiling for one-shot solutions is intentional -- the step cost (-0.1) discourages infinite iteration, so even a perfect single-step patch cannot reach 1.0. In practice this means:

• —Evaluation use case: PatchBench best differentiates models in the 0.40-0.94 band, where multi-step reasoning matters.
• —Training use case: The dense reward curve across medium_01's trajectory (0.44 x 7 then 0.94) is precisely the signal an RL fine-tuning loop can exploit to teach iterative debugging behavior.

A future version will introduce cross-file patches and tighter step budgets to push more tasks into the training-signal band for frontier models.

Architecture

patchbench/
├── inference.py              Baseline inference script (OpenAI client)
├── openenv.yaml              OpenEnv metadata
├── Dockerfile                Container build
├── pyproject.toml
├── requirements.txt
├── README.md
├── patchbench/               Environment package
│   ├── __init__.py
│   ├── models.py             Pydantic Observation / Action
│   ├── environment.py        PatchBenchEnv: reset / step / state
│   ├── grader.py             Subprocess-sandboxed pytest grader
│   └── tasks/                9 hand-crafted task scenarios
│       ├── easy/
│       ├── medium/
│       └── hard/
└── server/
    └── app.py                FastAPI HTTP wrapper for HF Space

Design Notes

Sandboxing. The grader writes patched code to a fresh tempfile.TemporaryDirectory() and runs pytest as a subprocess with a 15-second timeout. Timeouts, crashes, and parse failures are all caught -- the grader never raises to the caller.

Determinism. Baseline pass/fail sets for every task are pre-computed and stored in task.json. The same action always produces the same reward.

Efficiency. No task takes more than 12 steps. With three tasks and a typical LLM latency, the full inference script completes well under the 20-minute budget.

License

MIT