datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SWE-bench_Lite
Dataset Summary
SWE-bench Lite is subset of SWE-bench, a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Want to run inference now?
This dataset only contains the… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite.SWE-bench_Lite
Dataset Summary
SWE-bench Lite is subset of SWE-bench, a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Want to run inference now?
This dataset only contains the… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-bench_Lite.SWE-bench_Lite_filteredSWE-bench_Lite_filtered_1SWE-Bench-LiteSWE-bench_Lite_oracle
Dataset Summary
SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
This dataset SWE-bench_Lite_oracle includes a formatting of each instance using the "Oracle" retrieval… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite_oracle.SWE-bench_Lite-code-searchSWE-bench_Lite_bm25_27K
Dataset Summary
SWE-bench Lite is subset of SWE-bench, a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
This dataset SWE-bench_Lite_bm25_27K includes a formatting of each instance… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite_bm25_27K.SWE-bench_Lite_bm25_13K
Dataset Summary
SWE-bench Lite is subset of SWE-bench, a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
This dataset SWE-bench_Lite_bm25_13K includes a formatting of each instance… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite_bm25_13K.SWE-bench_Lite_UnstableSWE-bench_Liteswe-bench-lite-agent-traces-v14
AgentBRANE SWE-bench Lite Agent Traces v14
This release contains the 1,890 harness-native agent traces selected by the sealed SWE-bench Lite v14 publication record (1,379/1,890 resolved, 73.0%). It includes Claude Code, Codex, and Pi sessions across seven models and three replicates. No internal research notes are included.
Load the observation table:
from datasets import load_dataset
traces = load_dataset("melissapan/swe-bench-lite-agent-traces-v14", split="train")
Each row… See the full description on the dataset page: https://huggingface.co/datasets/melissapan/swe-bench-lite-agent-traces-v14.swe-bench-lite3
Dataset Card for "swe-bench-lite3"
More Information needed
swebenchlite_deleteSWE-bench_Verified_Lite_Annt
Dataset details
Appended difficulty annotations provided by OpenAI here
SWE-bench_Lite
Dataset Summary
SWE-bench Lite is subset of SWE-bench, a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 300 test Issue-Pull Request pairs from 11 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Want to run inference now?
This dataset only contains the… See the full description on the dataset page: https://huggingface.co/datasets/Luwayy/SWE-bench_Lite.SWEbenchLiteRR
SWEbenchLiteRR
An MTEB dataset
Massive Text Embedding Benchmark
Software Issue Localization.
Task category
t2t
Domains
Programming, Written
Reference
https://www.swebench.com/Source datasets:
tarsur909/mteb-swe-bench-lite-reranking
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_task("SWEbenchLiteRR")
evaluator = mteb.MTEB([task])
model = mteb.get_model(YOUR_MODEL)… See the full description on the dataset page: https://huggingface.co/datasets/mteb/SWEbenchLiteRR.SWE-bench_Lite_Dev_ExtendedSWE-Bench_Lite-logsSWE-bench_LiteSWE-bench_Lite_oracle_32kimport datasets
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
ds = datasets.load_dataset("princeton-nlp/SWE-bench_Lite_oracle", split="test")
def count_tokens(text):
return len(tokenizer.encode(text))
ds = ds.map(lambda x: {"num_tokens": count_tokens(x["text"])}, num_proc=10)
ds = ds.filter(lambdax: x["num_tokens"] <= 32_000)
SWE-bench_LiteSWE-Bench_Lite_UTBoostDataset Summary
In this dataset, we replace some test suites in princeton-nlp/SWE-bench_Verified with augmented test cases to enable a more rigorous evaluation of SWE-Bench. UTBoost was accepted in ACL 2025, and we have opened-sourced the code and metadata in https://github.com/CUHK-Shenzhen-SE/UTBoost.
Dataset Structure
An example of a SWE-bench datum is as follows:
instance_id: (str) - A formatted instance identifier, usually as repo_owner__repo_name-PR-number.
patch: (str) - The gold patch… See the full description on the dataset page: https://huggingface.co/datasets/Bertsekas/SWE-Bench_Lite_UTBoost.swe_bench_Lite_p_agentswe_doc_gen_SWE-bench_Lite_testSWE-bench_Lite-locagentSWE-bench_Lite_oracle_easyfrom datasets import load_dataset
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")
ds = load_dataset("princeton-nlp/SWE-bench_Verified", split="test")
ds_lite = load_dataset("princeton-nlp/SWE-bench_Lite_oracle", split="test")
def count_tokens(text):
return len(tokenizer.encode(text))
ds_easy = ds.filter(lambda x: x["difficulty"] == "<15 min fix")
ds_easy_lite = ds_lite.filter(lambda x: x["instance_id"] in ds_easy["instance_id"])… See the full description on the dataset page: https://huggingface.co/datasets/rasdani/SWE-bench_Lite_oracle_easy.SWE-bench_Lite_bm25_50k
SWE-bench_Lite
Generated via https://github.com/SWE-bench/SWE-bench/blob/main/swebench/inference/make_datasets/README.md
Set k=40 to generated the bm25 search results first, and then use both k=40 and max_context_len=50000 to generate the dataset.
Patch file recall: 64.67; higher than the official princeton-nlp/SWE-bench_Lite_bm25_13K (recall=33.67) and princeton-nlp/SWE-bench_Lite_bm25_27K (recall=49.00)
SWE-bench_Lite_bm25_500kSWE-bench_Lite
Generated via https://github.com/SWE-bench/SWE-bench/blob/main/swebench/inference/make_datasets/README.md
Set k=100 to generated the bm25 search results first, and then use both k=100 and max_context_len=500000 to generate the dataset.
Patch file recall: 90.67; higher than the official princeton-nlp/SWE-bench_Lite_bm25_13K (recall=33.67) and princeton-nlp/SWE-bench_Lite_bm25_27K (recall=49.00)
SWE-bench_Lite_bm25_100k
SWE-bench_Lite
Generated via https://github.com/SWE-bench/SWE-bench/blob/main/swebench/inference/make_datasets/README.md
Set k=40 to generated the bm25 search results first, and then use both k=40 and max_context_len=100000 to generate the dataset.
Patch file recall: 73.33; higher than the official princeton-nlp/SWE-bench_Lite_bm25_13K (recall=33.67) and princeton-nlp/SWE-bench_Lite_bm25_27K (recall=49.00)