nebius/SWE-rebench-leaderboard

Dataset Summary

❗❗❗ Please use Harbour Hub for the July 2026 evaluation split:https://hub.harborframework.com/datasets/ibragim-badertdinov/swe-rebench-07-2026/latest

SWE-rebench-leaderboard is a continuously updated, curated subset of the full SWE-rebench corpus, tailored for benchmarking software engineering agents on real-world tasks. These tasks are used in the SWE-rebench leaderboard. For more details on the benchmark methodology and data collection process, please refer to our paper SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents.

All Docker images required to run the tasks are pre-built and publicly available on Docker Hub. You do not need to build them yourself. The specific image for each task is listed in the docker_image column.

To get the exact subset of tasks used for a specific month's SWE-rebench-leaderboard, you can filter the dataset by the created_at field.

News

[2025/09/19] Added a split for each month. [2025/09/01] Added 52 August tasks, each with a corresponding Docker image. [2025/08/04] Added 34 July tasks, each with a corresponding Docker image.

How to Use

from datasets import load_dataset
ds = load_dataset('nebius/SWE-rebench-leaderboard')
ds_june_2025 = ds['test'].filter(lambda x: x['created_at'].startswith('2025-06'))

Dataset Structure

The SWE-rebench dataset schema extends the original SWE-bench schema with additional fields to support richer analysis. The complete schema is detailed in the table below. For more information about this data and methodology behind collecting it, please refer to our paper.

To execute tasks from SWE-rebench (i.e., set up their environments, apply patches, and run tests), we provide a fork of the original SWE-bench execution framework, adapted for our dataset's structure and features. The primary modification introduces functionality to source environment installation constants directly from the install_config field present in each task instance within SWE-rebench. This allows for more flexible and task-specific environment setups.

You can find the details of this modification in the following commit

To build the necessary Docker images and run agents on SWE-rebench tasks, you have two main options:

1. 1.Use our SWE-bench fork directly: Clone the fork and utilize its scripts for building images and executing tasks. The framework will automatically use the install_config from each task.
2. 2.Integrate similar functionality into your existing codebase: If you have your own execution framework based on SWE-bench or a different system, you can adapt it by implementing a similar mechanism to parse and utilize the install_config field from the SWE-rebench task instances. The aforementioned commit can serve as a reference for this integration.

License

The dataset is licensed under the Creative Commons Attribution 4.0 license. However, please respect the license of each specific repository on which a particular instance is based. To facilitate this, the license of each repository at the time of the commit is provided for every instance.

Citation

@misc{badertdinov2025swerebenchautomatedpipelinetask,
      title={SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents}, 
      author={Ibragim Badertdinov and Alexander Golubev and Maksim Nekrashevich and Anton Shevtsov and Simon Karasik and Andrei Andriushchenko and Maria Trofimova and Daria Litvintseva and Boris Yangel},
      year={2025},
      eprint={2505.20411},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2505.20411}
}