datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SWE-bench_VerifiedDataset Summary
SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process.
The dataset collects 500 test Issue-Pull Request pairs from popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The original… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified.SWE-bench_VerifiedDataset Summary
SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process.
The dataset collects 500 test Issue-Pull Request pairs from popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The original… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-bench_Verified.SWE-Bench-Verified-O1-reasoning-high-results
SWE-Bench Verified O1 Dataset
Executive Summary
This repository contains verified reasoning traces from the O1 model evaluating software engineering tasks. Using OpenHands + CodeAct v2.2, we tested O1's bug-fixing capabilities on the SWE-Bench Verified dataset, achieving a 28.8% success rate across 500 test instances.
Overview
This dataset was generated using the CodeAct framework, which aims to improve code generation through enhanced action-based reasoning.… See the full description on the dataset page: https://huggingface.co/datasets/AlexCuadron/SWE-Bench-Verified-O1-reasoning-high-results.SWE-Bench-Verifiedswebench-verified-sample-100-Qwen3-30B-evalswe-bench-verified-miniSWEBench-verified-mini is a subset of SWEBench-verified that uses 50 instead of 500 datapoints, requires 5GB instead of 130GB of storage and has approximately the same distribution of performance, test pass rates and difficulty as the original dataset.
You can find more details here: https://github.com/mariushobbhahn/make_swe_bench_verified_mini
If you use the Inspect implementation, you can merely switch the dataset: str = "princeton-nlp/SWE-bench_Verified", to dataset: str =… See the full description on the dataset page: https://huggingface.co/datasets/MariusHobbhahn/swe-bench-verified-mini.SWE-bench_Not_VerifiedSWE-Bench-Verified-O1-native-tool-calling-reasoning-high-results
SWE-Bench Verified O1 Dataset
Executive Summary
This repository contains verified reasoning traces from the O1 model evaluating software engineering tasks. Using OpenHands + CodeAct v2.2, we tested O1's bug-fixing capabilities using their native tool calling capabilities on the SWE-Bench Verified dataset, achieving a 45.8% success rate across 500 test instances.
Overview
This dataset was generated using the CodeAct framework, which aims to improve code… See the full description on the dataset page: https://huggingface.co/datasets/AlexCuadron/SWE-Bench-Verified-O1-native-tool-calling-reasoning-high-results.SWE-bench_Verified-code-searchswe-bench-verified-codebase-content-staging
SWE-Bench Verified
import argparse
from dataclasses import dataclass, asdict
import datasets
from pathlib import Path
import subprocess
from typing import Dict, List
import tqdm
from datasets import Dataset
import hashlib
from dataclasses import dataclass
@dataclass
classCodebaseFile:
path: str
content: str
class SWEBenchProblem:
def __init__(self, row):
self._row = row
@property
def repo(self) -> str:
return self._row["repo"]… See the full description on the dataset page: https://huggingface.co/datasets/ScalingIntelligence/swe-bench-verified-codebase-content-staging.swe-bench-verified-codebase-content
SWE-Bench Verified Codebase Content Dataset
Introduction
SWE-bench is a popular benchmark that measures how well systems can solve real-world software engineering problems. To solve SWE-bench problems, systems need to interact with large codebases that have long commit histories. Interacting with these codebases in an agent loop using git naively can be slow, and the repositories themselves take up large amounts of storage space.
This dataset provides the complete Python… See the full description on the dataset page: https://huggingface.co/datasets/ScalingIntelligence/swe-bench-verified-codebase-content.CoderForge-Preview-32B-SWE-Bench-Verified-Evaluation-trajectoriesSWE-bench_Verified-locagentSWE-bench_Verified_commit0Lego-RL-SWE-Bench-Verified
Lego-RL-SWE-Bench-Verified
The 500 SWE-bench Verified instances as ready-to-run harbor RL environments —
the exact evaluation set behind every SWE-bench Verified number in
LEGO-RL, packaged the same way as the training
set Lego-X/Lego-RL-2699 so
one trainer reads both.
Two parallel views of the same 500 instances:
View
Path
What it is
Official SWE-bench records
swebench_verified_official_500/
The upstream princeton-nlp/SWE-bench_Verified rows, verbatim
Harbor RL… See the full description on the dataset page: https://huggingface.co/datasets/Lego-X/Lego-RL-SWE-Bench-Verified.SWE-bench_Verified-discriminative
SWE-bench Verified Discriminative Subsets
Dataset Description
This dataset contains discriminative subsets of SWE-bench Verified designed to provide more sensitive evaluation of SWE-agent capabilities. As top-performing agents achieve 73%+ on the full benchmark, these subsets focus on truly challenging problems to better discriminate between cutting-edge systems.
Key Features
4 discriminative splits targeting different evaluation needs
335 carefully selected… See the full description on the dataset page: https://huggingface.co/datasets/jatinganhotra/SWE-bench_Verified-discriminative.SWE-bench_Verified_50
Dataset Summary
For faster evaluation, this dataset is generated by doing stratified sampling on difficulty annotations provided by OpenAI here
on the full verified dataset
Sampling script is also uploaded in the repo.
SWE-bench_Verified_oracleSWE-bench_Verified_OAI_Annt
Dataset details
Appended difficulty annotations provided by OpenAI here
SWEbenchVerifiedRR
SWEbenchVerifiedRR
An MTEB dataset
Massive Text Embedding Benchmark
Software Issue Localization for SWE-bench Verified
Task category
t2t
Domains
Programming, Written
Reference
https://openai.com/index/introducing-swe-bench-verified/
Source datasets:
tarsur909/mteb-swe-bench-verified-reranking
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/SWEbenchVerifiedRR.swebench_verified_random_100_folders_a3_rl_DCAgent_selfinstruct_naive_sandboxes_2_c2a08420swebench_verified_random_100_folders_a3_rl_DCAgent_inferredbugs_sandboxes_verifierb984e9c9SWE-bench_Verified_Lite_Annt
Dataset details
Appended difficulty annotations provided by OpenAI here
codebase-content-SWE-bench_Verified-with-comments-and-testsSWE-bench_Verified_With_Annotationsswebench_verified_GLM_4_7_swesmith_sandboxes_with_tests_oracle_verified_120s_mad3f88d27swebench-verified-v2swe-bench-verified-djangoswebench-verified-deepseek-v4-flash-failure-analysis
SWE-bench Verified runs & failure analysis — DeepSeek-V4-flash (local) × mini-swe-agent
Per-instance analysis of SWE-bench Verified runs of a locally-served DeepSeek-V4-flash model
driven by mini-swe-agent, graded with the official
SWE-bench harness. Each instance carries the full agent trajectory, a readable transcript, the
submitted patch, the harness test output, deterministic metrics, and a hand-verified qualitative
root-cause diagnosis.
Current numbers (resolve rates… See the full description on the dataset page: https://huggingface.co/datasets/daaain/swebench-verified-deepseek-v4-flash-failure-analysis.codebase-content-SWE-bench_Verified-no-comments-and-file-types