datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SWE-smith
SWE-smith Dataset
Code
•
Paper
•
Site
[12/14/2025] NOTE: We will no longer actively update this dataset.
While this dataset is still functional and usable, we recommend you use the `SWE-bench/SWE-smith-[lang]` datasets.
For better maintainability and ease-of-use, we are maintaining language-specific datasets in lieu of this mono-repo.
The SWE-smith Dataset is a training dataset of 50137 task instances from 128 GitHub repositories, collected using the SWE-smith toolkit.… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-smith.terminal-bench-2.0Warning: The leaderboard above is unofficial. The official leaderboard is https://www.tbench.ai/leaderboard/terminal-bench/2.0, in which entires are audited for correct configuration, results show which agent harness is used, and verified trajectories are publicly viewable.
Warning: The dataset is a read-only mirror. The primary source for this dataset is on GitHub: https://github.com/harbor-framework/terminal-bench-2. Please open issues and pull requests there.
How this mirror was created… See the full description on the dataset page: https://huggingface.co/datasets/harborframework/terminal-bench-2.0.SWE-smith-py
SWE-smith Dataset
Code
•
Paper
•
Site
As of 12/14/2025, SWE-smith: Python contains 50908 task instances from 131 GitHub repositories
The SWE-smith Dataset is the largest open source dataset for training software engineering agents.
All SWE-smith task instances come with an executable environment.
To learn more about how to use this dataset to train Language Models for Software Engineering, please refer to the documentation.
big_bench_hardAll rights and obligations of the dataset are with original authors of the paper/dataset.
I have merely made this dataset with a MIT licence available on HuggingFace.
BIG-Bench Hard Dataset
This repository contains a copy of the BIG-Bench Hard dataset.
Small edits to the formatting of the dataset are made to integrate it into the Inspect Evals repository, a community contributed LLM
evaulations for Inspect AI a framework by the UK AI Safety Institute.
The BIG-Bench Hard dataset is a… See the full description on the dataset page: https://huggingface.co/datasets/Joschka/big_bench_hard.ALE-Bench
ALE-Bench
Dataset Description
ALE-Bench is a benchmark for evaluating AI systems on score-based algorithmic programming contests.
This dataset is officially provided by AtCoder Inc..
Please be sure to check the "License" section below.
Please read our blog post and our paper for more details.
Related resources:
Preprint paper (arXiv)
Sakana AI Blog (English)
Sakana AI Blog (Japanese)
GitHub repository
Leaderboard
Usage
Our Python library automatically… See the full description on the dataset page: https://huggingface.co/datasets/SakanaAI/ALE-Bench.SWE-smith-trajectories
SWE-smith Trajectories
Code
•
Paper
•
Site
This dataset contains the 5017 trajectories we fine-tuned Qwen 2.5 Coder Instruct on, leading to
SWE-agent-LM-32B, a coding LM agent that
achieve 40.2% on SWE-bench Verified (no verifiers or multiple rollouts, just 1 attempt per instance).
Trajectories were generated by running SWE-agent + Claude 3.7 Sonnet on task instances from
the SWE-smith dataset.
Multi-SWE-bench
👋 Overview
This repository contains the Multi-SWE-bench dataset, introduced in Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving, to address the lack of multilingual benchmarks for evaluating LLMs in real-world code issue resolution.
Unlike existing Python-centric benchmarks (e.g., SWE-bench), this framework spans 7 languages (Java, TypeScript, JavaScript, Go, Rust, C, and C++) with 1,632 high-quality instances,
curated from 2,456 candidates by 68 expert… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench.Long-Horizon-Terminal-Bench
Long-Horizon Terminal-Bench (LHTB)
LHTB is a 46-task benchmark for measuring how well LLM agents sustain useful
work in a containerized terminal over hundreds of steps. Unlike short-horizon
coding benchmarks where an agent writes one artifact and stops, LHTB drops the agent
into a stateful environment and grades it with hidden, rebuild-from-artifact
verifiers — self-reported progress does not count.
📝 Blog: https://zli12321.github.io/LHTB/
🏆 Leaderboard:… See the full description on the dataset page: https://huggingface.co/datasets/IntelligenceLab/Long-Horizon-Terminal-Bench.Multi-SWE-bench_trajs
🧠 Multi-SWE-bench Trajectories
This repository stores all trajectories and logs generated by agents evaluated on the Multi-SWE-bench leaderboard.
📚 Citation
@misc{zan2025multiswebench,
title={Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving},
author={Daoguang Zan and Zhirong Huang and Wei Liu and Hanwu Chen and Linhao Zhang and Shulin Xin and Lu Chen and Qi Liu and Xiaojian Zhong and Aoyan Li and Siyao Liu and Yongsheng Xiao and Liangqiang Chen… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench_trajs.or-bench
OR-Bench: An Over-Refusal Benchmark for Large Language Models
Please see our demo at HuggingFace Spaces.
Overall Plots of Model Performances
Below is the overall model performance. X axis shows the rejection rate on OR-Bench-Hard-1K and Y axis shows the rejection rate on OR-Bench-Toxic. The best aligned model should be on the top left corner of the plot where the model rejects the most number of toxic prompts and least number of safe prompts. We also plot a blue line… See the full description on the dataset page: https://huggingface.co/datasets/bench-llm/or-bench.Multi-SWE-bench
Multi-SWE-bench
Re-upload of ByteDance's Multi-SWE-bench
evaluation benchmark: 2,132 issue-resolving tasks across the seven Multi-SWE languages.
This is the held-out eval benchmark; for RL training data use
PrimeIntellect/Multi-SWE-RL-Verified.
Changes vs upstream
Storage schema only: per-test maps are stored as columnar struct-of-lists so the rows load
cleanly with datasets. Row content is unchanged.
License mirrors upstream: ByteDance licenses the dataset… See the full description on the dataset page: https://huggingface.co/datasets/PrimeIntellect/Multi-SWE-bench.explicit-edit-benchmark
Explicit Edit Benchmark
226 deterministic exact-edit tasks, run by different agents, harnesses, models and configurations. Every observation records what the harness did and whether the resulting files matched byte for byte.
Source code and benchmark runner: GitHub — Explicit Edit Benchmark
Open the interactive Explorer to compare agents, harnesses, models, versions, reasoning modes, correctness, recovery, time, cost and tokens.
Leaderboard by model route
Score v2… See the full description on the dataset page: https://huggingface.co/datasets/alexshpunt/explicit-edit-benchmark.car-bench-dataset
CAR-Bench Dataset
CAR-Bench is a benchmark for evaluating AI voice assistants in a realistic automotive (car) environment.
It tests an agent's ability to correctly use vehicle control tools, handle disambiguation, and avoid hallucinations.
Dataset Structure
The dataset is organized into task configs and mock data configs:
Tasks
Each task defines a user persona, an instruction, the initial vehicle/environment context, and the ground-truth sequence of tool-call… See the full description on the dataset page: https://huggingface.co/datasets/johanneskirmayr/car-bench-dataset.air-bench-2024
AIRBench 2024
AIRBench 2024 is a AI safety benchmark that aligns with emerging government
regulations and company policies. It consists of diverse, malicious prompts
spanning categories of the regulation-based safety categories in the
AIR 2024 safety taxonomy.
Dataset Details
Dataset Description
AIRBench 2024 is a AI safety benchmark that aligns with emerging government
regulations and company policies. It consists of diverse, malicious prompts
spanning… See the full description on the dataset page: https://huggingface.co/datasets/stanford-crfm/air-bench-2024.chi-bench
Clinical Healthcare In-Situ Environment
Task fixtures for a long-horizon, policy-rich healthcare-workflow agent benchmark
What is in this dataset
CHI-Bench evaluates AI agents on end-to-end U.S. healthcare workflows across three long-horizon domains: provider prior authorization, payer utilization management, and population care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed over MCP, with a 1… See the full description on the dataset page: https://huggingface.co/datasets/actava/chi-bench.SWE-Bench-Verified-O1-reasoning-high-results
SWE-Bench Verified O1 Dataset
Executive Summary
This repository contains verified reasoning traces from the O1 model evaluating software engineering tasks. Using OpenHands + CodeAct v2.2, we tested O1's bug-fixing capabilities on the SWE-Bench Verified dataset, achieving a 28.8% success rate across 500 test instances.
Overview
This dataset was generated using the CodeAct framework, which aims to improve code generation through enhanced action-based reasoning.… See the full description on the dataset page: https://huggingface.co/datasets/AlexCuadron/SWE-Bench-Verified-O1-reasoning-high-results.RoadmapBench
RoadmapBench
A benchmark for evaluating AI coding agents on multi-target, long-horizon software development tasks derived from open-source project version upgrades.
Overview
RoadmapBench contains 115 tasks spanning 17 open-source repositories across 5 programming languages (Python, TypeScript, Go, Rust, C++). Each task requires an agent to implement multiple interdependent features that correspond to a real version upgrade of the target project.
Task Structure… See the full description on the dataset page: https://huggingface.co/datasets/benchmark-anon-2026/RoadmapBench.skillsbenchWarning: The leaderboard above is generated by Hugging Face eval-results and may be incomplete until evaluation_framework: benchflow is accepted and deployed. The audited SkillsBench v1.1 result archive is https://huggingface.co/datasets/benchflow/skillsbench-leaderboard, with all retained submissions normalized under submissions/skillsbench/v1.1/ and compact official exports under leaderboard/skillsbench/v1.1/.
Warning: The dataset is a read-only mirror. The primary source for this benchmark… See the full description on the dataset page: https://huggingface.co/datasets/benchflow/skillsbench.migration-bench-java-full
MigrationBench
1. 📖 Overview
🤗 MigrationBench
is a large-scale code migration benchmark dataset at the repository level,
across multiple programming languages.
Current and initial release includes java 8 repositories with the maven build system… See the full description on the dataset page: https://huggingface.co/datasets/AmazonScience/migration-bench-java-full.Multi-SWE-bench_mini
👋 Overview
To make the benchmark more lightweight and minimize resource consumption, we have introduced the Multi-SWE-bench mini.
This version contains 400 instances, with 50 instances per language (Python, Java, TypeScript, JavaScript, Go, Rust, C, and C++). It covers a range of difficulty levels—easy, medium, and hard—ensuring a balanced and efficient evaluation across multiple languages without excessive resource usage.The leaderboard can be found at:… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench_mini.SWE-smith-go
SWE-smith Dataset
Code
•
Paper
•
Site
As of 12/14/202, SWE-smith: Golang contains 8212 task instances from 87 GitHub repositories
The SWE-smith Dataset is the largest open source dataset for training software engineering agents.
All SWE-smith task instances come with an executable environment.
To learn more about how to use this dataset to train Language Models for Software Engineering, please refer to the documentation.
terminal-bench
Terminal-Bench Dataset
This dataset contains tasks from Terminal-Bench, a benchmark for evaluating AI agents in real terminal environments. Each task is packaged as a complete, self-contained archive that preserves the exact directory structure, binary files, Docker configurations, and test scripts needed for faithful reproduction.
The archive column contains a gzipped tarball of the entire task directory.
Dataset Overview
Terminal-Bench evaluates AI agents on… See the full description on the dataset page: https://huggingface.co/datasets/ia03/terminal-bench.DeepReview-Bench
DeepReview-Bench
A benchmark package built from DeepReview-13K. Each completed paper directory
contains the selected review-time PDF, a Markdown conversion generated from the
DeepReview-13K embedded source text, human-review data, and provenance metadata.
Contents
papers/<paper_id>/:
file
description
paper.pdf
selected PDF revision for review-time use
paper.md
Markdown converted from DeepReview-13K embedded source text
paper.source.tex
embedded… See the full description on the dataset page: https://huggingface.co/datasets/cmwqfcmwqf/DeepReview-Bench.DTap-Bench-Agent-Trajectories
DecodingTrust-Agent Platform
A Controllable and Interactive Red-Teaming Platform for AI Agents.
This is the full collection of the agent trajectories produced from evaluating the DTap-Bench from DecodingTrust-Agent Platform (DTAP),
spanning 14 real-world domains and 50+ simulation environments that replicate widely-used
systems such as Google Workspace, PayPal, Slack, Salesforce, Snowflake, and Databricks. Each task
ships the configuration the evaluator needs to spin up the… See the full description on the dataset page: https://huggingface.co/datasets/AI-Secure/DTap-Bench-Agent-Trajectories.Claw-SWE-Bench
Claw-SWE-Bench
Paper: Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-Style Agent Harnesses on Coding Tasks
A multilingual issue-resolving benchmark with two evaluation configs:
full — 350 instances (300 from SWE-bench Multilingual + 50 Python from
SWEBench-verified-mini's size_optimized_sample).
lite — 80-instance calibrated subset (10 per language across 8
languages: Java, Go, Rust, JS/TS, C/C++, Ruby, PHP, Python). Designed for
low-cost iteration on harness… See the full description on the dataset page: https://huggingface.co/datasets/TokenRhythm/Claw-SWE-Bench.terminal-bench-pro
Terminal-Bench Pro
Overview
Terminal-Bench Pro is a systematic extension of the original Terminal-Bench, designed to address key limitations in existing terminal-agent benchmarks.
400 tasks (200 public + 200 private) across 8 domains: data processing, games, debugging, system admin, scientific computing, software engineering, ML, and security
Expert-designed tasks derived from real-world scenarios and GitHub issues
High test coverage with ~28.3 test cases per… See the full description on the dataset page: https://huggingface.co/datasets/alibabagroup/terminal-bench-pro.Multi-SWE-bench-flash
👋 Overview
This repository contains the Multi-SWE-bench dataset, introduced in Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving, to address the lack of multilingual benchmarks for evaluating LLMs in real-world code issue resolution.
Unlike existing Python-centric benchmarks (e.g., SWE-bench), this framework spans 7 languages (Java, TypeScript, JavaScript, Go, Rust, C, and C++) with 1,632 high-quality instances,
curated from 2,456 candidates by 68 expert annotators… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench-flash.panda-bench
PandaBench
PandaBench is a comprehensive benchmark for evaluating Large Language Model (LLM) safety, focusing on jailbreak attacks, defense mechanisms, and evaluation methodologies.
The PandaGuard framework architecture illustrating the end-to-end pipeline for LLM safety evaluation. The system connects three key components: Attackers, Defenders, and Judges.
Dataset Description
This repository contains the benchmark results from extensive evaluations of various… See the full description on the dataset page: https://huggingface.co/datasets/Beijing-AISI/panda-bench.physical-ai-bench-generation
Physical AI Bench - Generation
Paper | Code
Dataset Description
The PAI-Bench is a benchmark to measure the progress of world models quantitatively.
The predict task contains a list of 1044 samples of text prompts, conditioning images, and qa pairs, covering Physical AI target domains including autonomous vehicle (AV) driving, robotics, industry (smart space), physics, human, and common sense. All the questions are binary questions, and the answer is either Yes or No. Our… See the full description on the dataset page: https://huggingface.co/datasets/shi-labs/physical-ai-bench-generation.Data-Prep-Bench
Data-Prep-Bench
This repository contains the data presented in DataPrep-Bench: Benchmarking LLMs as Training Data Preparators.
Code: https://github.com/OpenDCAI/Data-Preparation-Bench
Dataset Overview
This dataset is a comprehensive resource built for Supervised Fine-Tuning (SFT) and evaluation of Large Language Models (LLMs), covering six domains: Finance, Medicine, Law, Mathematics, Science, and General.
A key feature of this dataset is that we employed 12… See the full description on the dataset page: https://huggingface.co/datasets/lhpku20010120/Data-Prep-Bench.