UnipatAI/Monthly-SWEBench-2026-03

Monthly-SWEBench-2026-03

Monthly-SWEBench-2026-03 is a curated benchmark of 112 real-world software engineering tasks, sourced from GitHub pull requests merged in March 2026. Tasks are in Harbor format and can be run with any Harbor-compatible agent.

View leaderboard and results →

• —112 tasks — 68 bugfix + 44 non-bugfix
• —Tasks span diverse open-source repositories
• —Each task includes a runnable environment, test suite, and reference solution

Task Structure

Each task is a self-contained directory:

<task_id>/
├── task.toml        # Task metadata and resource limits
├── instruction.md   # Natural-language description given to the agent
├── environment/     # Dockerfile + setup scripts defining the starting state
├── tests/           # Verifier scripts run after the agent finishes
└── solution/        # Reference solution (ground-truth patch)

`task.toml` specifies the task name, difficulty, category, and execution constraints — agent timeout, verifier timeout, CPU/memory/disk limits for the container.

`instruction.md` is what the agent sees. It describes what needs to be implemented or fixed, the expected behavior, and acceptance criteria — written as a natural language engineering spec, not as a hint or walkthrough.

`environment/` contains the Dockerfile (and any supporting files) that builds the container the agent works inside. The environment captures the exact repository state before the PR was merged — the agent starts from a broken or incomplete codebase.

`tests/` contains the verifier. After the agent finishes, Harbor runs the test script inside the container. The script writes 1.0 to /logs/verifier/reward.txt if the task passes, 0.0 if it fails. Tests are derived from the actual CI checks in the original PR.

`solution/` contains the reference patch — the actual code change from the merged PR. This is used by the oracle agent and can be used for debugging or analysis.

Run with Harbor

Install

pip install harbor

Download

huggingface-cli download UnipatAI/Monthly-SWEBench-2026-03 \
  --repo-type dataset --local-dir ./monthly-swebench-2026-03

cd monthly-swebench-2026-03
tar -I zstd -xf bugfix.tar.zst
tar -I zstd -xf non_bugfix.tar.zst

Run

export ANTHROPIC_API_KEY=<your-key>

# Run bugfix tasks
harbor run \
  -p ./monthly-swebench-2026-03/bugfix \
  --agent terminus-2 \
  --model anthropic/claude-opus-4-5 \
  --n-concurrent 4

# Run non-bugfix tasks
harbor run \
  -p ./monthly-swebench-2026-03/non_bugfix \
  --agent terminus-2 \
  --model anthropic/claude-opus-4-5 \
  --n-concurrent 4

terminus-2 is Harbor's built-in multi-turn agent and works with any model supported by LiteLLM. Swap --model to evaluate different models, or use --agent to try other agents (claude-code, codex, openhands, aider, etc.).

View results

harbor view

Citation

@dataset{monthly_swebench_2026_03,
  title  = {Monthly-SWEBench-2026-03},
  author = {UnipatAI},
  year   = {2026},
  url    = {https://huggingface.co/datasets/UnipatAI/Monthly-SWEBench-2026-03}
}