Northy1717/SWE-bench-Live

<p align="center"> <a href="http://swe-bench.github.io"> <img src="assets/banner.png" style="height: 10em" alt="swe-bench-live" /> </a> </p>

<p align="center"> <em>A brand-new, continuously updated SWE-bench-like dataset powered by an automated curation pipeline.</em> </p>

For the official data release page, please see [microsoft/SWE-bench-Live](https://github.com/microsoft/SWE-bench-Live).

Dataset Summary

SWE-bench-Live is a live benchmark for issue resolving, designed to evaluate an AI system’s ability to complete real-world software engineering tasks. Thanks to our automated dataset curation pipeline, we plan to update SWE-bench-Live on a monthly basis to provide the community with up-to-date task instances and support rigorous and contamination-free evaluation.

News

• —9/17/2025: Dataset updated! We’ve finalized the update process for SWE-bench-Live: Each month, we will add 50 newly verified, high-quality issues to the dataset. The lite and verified splits will remain frozen, ensuring fair leaderboard comparisons and keeping evaluation costs manageable. To access the latest issues, please refer to the full split!
• —07/19/2025: We've employed a LLM filter to automatically filter full dataset to create SWE-bench-Live Verified. The initial Verified subset contains 500 instances from 2024-07 to 2025-04.
• —06/30/2025: We’ve updated the dataset — it now includes a total of 1,565 task instances across 164 repositories!
• —05/21/2025: The initial release of SWE-bench-Live includes 1,319 latest (created after 2024) task instances, each paired with an instance-level Docker image for test execution, covering 93 repositories.

<p align="center"> <img src="assets/overview.png" alt="SWE-bench-Live Curation Pipeline" style="width: 100%; max-width: 800px;" /> <br> <em>SWE-bench-Live Curation Pipeline</em> </p>

Dataset Structure