datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SWE-bench_VerifiedDataset Summary
SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process.
The dataset collects 500 test Issue-Pull Request pairs from popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The original… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified.SWE-bench_VerifiedDataset Summary
SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process.
The dataset collects 500 test Issue-Pull Request pairs from popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
The original… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-bench_Verified.HLE-Verified
HLE-Verified
A Systematic Verification and Structured Revision of Humanity’s Last Exam
Overview
Humanity’s Last Exam (HLE) is a high-difficulty, multi-domain benchmark designed to evaluate advanced reasoning capabilities across diverse scientific and technical domains.
Following its public release, members of the open-source community raised concerns regarding the reliability of certain items. Community discussions and informal replication attempts suggested that some… See the full description on the dataset page: https://huggingface.co/datasets/skylenage-ai/HLE-Verified.terminal-bench-2-verified
Terminal-Bench 2.0 Verified: Instruction & Environment Fix Version
中文版本
2026.08.18 Update: Following the release of Terminal-Bench 2.1, we conducted another round of review on several tasks' grading and instructions, validating each fix end-to-end inside the actual task images. This round fixes 6 tasks in two categories: Grading/Test Fixes (dna-insert, make-doom-for-mips, filter-js-from-html, install-windows-3.11) — the test logic itself misjudged valid submissions or let… See the full description on the dataset page: https://huggingface.co/datasets/zai-org/terminal-bench-2-verified.distill_r1_qwen_math_1.5b_128_solns_math_verificationsHLE-Verified
HLE-Verified (HF-native JSONL)
This dataset is a lightweight, evaluation-ready reformatting of the HLE-Verified benchmark created by the Skylenage Team.
Original work: Weiqi Zhai et al., "HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam" (arXiv:2602.13964)
Original dataset: skylenage/HLE-Verified
Original repository: SKYLENAGE-AI/HLE-Verified
Source & Snapshot
Converted from skylenage/HLE-Verified snapshot… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab/HLE-Verified.NuminaMath-1.5-RL-Verifiable
Dataset Card for NuminaMath-1.5-RL-Verifiable
Dataset Summary
NuminaMath-1.5-RL-Verifiable is a curated subset of the NuminaMath-1.5 dataset, specifically filtered to support reinforcement learning applications requiring verifiable outcomes. This collection consists of 131,063 math word problems from the original dataset that meet strict filtering criteria: all problems have definitive numerical answers, validated problem statements and solutions, and come from… See the full description on the dataset page: https://huggingface.co/datasets/nlile/NuminaMath-1.5-RL-Verifiable.ubuntu_osworld_verified_trajs
OSWorld-Verified Model Trajectories
This repository contains trajectory results from various AI models evaluated on the OSWorld benchmark - a comprehensive evaluation environment for multimodal agents in real computer environments.
Dataset Overview
This dataset includes evaluation trajectories and results from multiple state-of-the-art models tested on OSWorld tasks.
File Structure
Each zip file contains complete evaluation trajectories including:… See the full description on the dataset page: https://huggingface.co/datasets/xlangai/ubuntu_osworld_verified_trajs.SWE-Bench-Verified-O1-reasoning-high-results
SWE-Bench Verified O1 Dataset
Executive Summary
This repository contains verified reasoning traces from the O1 model evaluating software engineering tasks. Using OpenHands + CodeAct v2.2, we tested O1's bug-fixing capabilities on the SWE-Bench Verified dataset, achieving a 28.8% success rate across 500 test instances.
Overview
This dataset was generated using the CodeAct framework, which aims to improve code generation through enhanced action-based reasoning.… See the full description on the dataset page: https://huggingface.co/datasets/AlexCuadron/SWE-Bench-Verified-O1-reasoning-high-results.Multi-SWE-RL-Verified
Multi-SWE-RL-Verified
Gold-patch-validated subset of
PrimeIntellect/Multi-SWE-RL-Reupload
(ByteDance's Multi-SWE-RL): 2,232 / 4,703 rows across
C, Go, Java, JavaScript, Rust, and TypeScript that produce a clean reward signal end-to-end.
Default dataset of the multiswe_v1 taskset.
Changes vs upstream
Starting from the 4,703-row re-upload:
C++ dropped wholesale — 0/449 rows passed gold-patch validation in pass 1; the images are
broken for scoring, not merely… See the full description on the dataset page: https://huggingface.co/datasets/PrimeIntellect/Multi-SWE-RL-Verified.swebench-verified-trajectories
SWE-bench Verified — mini-swe-agent trajectories (10 models)
Native .traj.json files from running mini-swe-agent on all 500 SWE-bench Verified instances
for each of the 10 models below. Downloaded from
Docent via preprocessing/swebench_docent.py in
contrastive_learning.
Layout
swebench_verified_raw/
<model_label>/
<instance_id>/
<instance_id>.traj.json
Per-file schema
{
"instance_id": str,
"trajectory_format": "mini-swe-agent-1.1"… See the full description on the dataset page: https://huggingface.co/datasets/tarsur385/swebench-verified-trajectories.Big-Math-RL-Verified
Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models
Big-Math is the largest open-source dataset of high-quality mathematical problems, curated specifically for reinforcement learning (RL) training in language models. With over 250,000 rigorously filtered and verified problems, Big-Math bridges the gap between quality and quantity, establishing a robust foundation for advancing reasoning in LLMs.
Request Early Access to Private… See the full description on the dataset page: https://huggingface.co/datasets/SynthLabsAI/Big-Math-RL-Verified.SWE-Bench-Verifiedswebench-verified-sample-100-Qwen3-30B-evalsoft-trigger-verifiedtau2-bench-verified-airline
tau2-bench-verified — airline domain (mirror)
Mirror of the airline domain from
amazon-agi/tau2-bench-verified
(MIT License), pinned at commit 864350a8971a8f8ee9e7b8472e2edc380a806b0c.
Re-hosted for the OpenRouter native TypeScript benchmark harness so it can fetch
the verified airline tasks + environment DB at runtime.
Contents
tasks/test.jsonl — 50 verified airline tasks. Each row has a single
task_json string column holding one verbatim tau2 v2 task object
(id… See the full description on the dataset page: https://huggingface.co/datasets/abhinavpola/tau2-bench-verified-airline.epoch_ai_swebench_verified
Epoch AI SWE-bench Verified Traces
Complete public trace archives and an analysis-ready Parquet conversion of Epoch AI's SWE-bench Verified evaluations.
Contents
34 published evaluation runs covering 16,456 traces (484 SWE-bench instances per run).
data/: loadable Parquet data, one exact trace per row.
original/: the byte-identical .eval archives published by Epoch AI.
run_metadata/: non-sample files from each .eval archive (header.json, summaries, reductions… See the full description on the dataset page: https://huggingface.co/datasets/naderalfares/epoch_ai_swebench_verified.a11oy-verifiable-corpus
Part of the SZL Holdings governed estate — claims are designed to carry checkable receipts. Verification proves integrity & origin, never accuracy or performance.
a11oy — Verifiable Corpus · verify it yourself
This dataset publishes a11oy's signed receipts and proof surface so that
anyone can independently verify them — no trust in SZL Holdings required.
Every receipt here carries the full cryptographic material needed to check its
signature offline;… See the full description on the dataset page: https://huggingface.co/datasets/SZLHOLDINGS/a11oy-verifiable-corpus.Scale-SWE-Verified
Scale-SWE-Verified
Gold-patch-validated fork of
AweAI-Team/Scale-SWE
(paper): 17,202 / 20,181 Python issue-resolving tasks
that produce a clean reward signal end-to-end. Default dataset of the scaleswe_v1 taskset.
Changes vs upstream
Validation (ours) removed 2,979 / 20,181 rows (14.8%):
892 rows whose image_url appears in
scale-swe-exclude-images.json.
2,061 rows categorized gold_patch_failure in
scale-swe-validation.jsonl.
15 rows categorized noop_pass… See the full description on the dataset page: https://huggingface.co/datasets/PrimeIntellect/Scale-SWE-Verified.R2E-Gym-Subset-Verified
R2E-Gym-Subset-Verified
Gold-patch-validated subset of
R2E-Gym/R2E-Gym-Subset
(paper). The train split contains
4,522 / 4,578 rows (98.78%) verified scoreable end-to-end: apply the gold patch, run the
upstream /testbed/run_tests.sh baked into the row's image, check the parsed outcomes against
expected_output_json.
Changes vs upstream
Validation-only subset — our passes, run in fresh sandboxes per row: one full pass at
concurrency 200, then a 10× retry pass over… See the full description on the dataset page: https://huggingface.co/datasets/PrimeIntellect/R2E-Gym-Subset-Verified.SWE-PolyBench_Verified
SWE-PolyBench
SWE-PolyBench is a multi language repo level software engineering benchmark. Currently it includes 4 languages: Python, Java, Javascript, and Typescript. The number of instances in the verified split is:
Javascript: 100
Typescript: 100
Python: 113
Java: 69
Datasets
There are total three datasets available under SWE-PolyBench. AmazonScience/SWE-PolyBench is the full dataset, AmazonScience/SWE-PolyBench_500 is the stratified sampled dataset with 500 instances… See the full description on the dataset page: https://huggingface.co/datasets/AmazonScience/SWE-PolyBench_Verified.verifiable-coding-problems
SYNTHETIC-1
This is a subset of the task data used to construct SYNTHETIC-1. You can find the full collection here
verifiable-coding-problems-python
Dataset Card for Verifiable Coding Problems Python 10k
This dataset contains all Python problems from PrimeIntellect's verifiable-coding-problems dataset. We have formatted the verification_info and metadata columns to be proper dictionaries, but otherwise the data is the same. Please see their dataset for more details.
simpleqa-verified
SimpleQA Verified
A 1,000-prompt factuality benchmark from Google DeepMind and Google Research, designed to reliably evaluate LLM parametric knowledge.
▶ SimpleQA Verified Leaderboard on Kaggle▶ Technical Report▶ Evaluation Starter Code
Benchmark
SimpleQA Verified is a 1,000-prompt benchmark for reliably evaluating Large Language Models (LLMs) on short-form factuality
and parametric knowledge. The authors from Google DeepMind and Google Research… See the full description on the dataset page: https://huggingface.co/datasets/google/simpleqa-verified.VeriContest
VeriContest
This dataset accompanies the paper "VeriContest: A Competitive-Programming
Benchmark for Verifiable Code Generation"
(arXiv:2605.08553).
VeriContest is a benchmark of 946
competitive-programming problems from LeetCode and Codeforces for verifiable
code generation in Rust with Verus.
Each problem contains a natural-language problem description, formal
specification, Rust code, proofs, positive and negative test cases, and
metadata. The benchmark supports both isolated… See the full description on the dataset page: https://huggingface.co/datasets/Gax-c/VeriContest.swe-bench-verified-miniSWEBench-verified-mini is a subset of SWEBench-verified that uses 50 instead of 500 datapoints, requires 5GB instead of 130GB of storage and has approximately the same distribution of performance, test pass rates and difficulty as the original dataset.
You can find more details here: https://github.com/mariushobbhahn/make_swe_bench_verified_mini
If you use the Inspect implementation, you can merely switch the dataset: str = "princeton-nlp/SWE-bench_Verified", to dataset: str =… See the full description on the dataset page: https://huggingface.co/datasets/MariusHobbhahn/swe-bench-verified-mini.SWEbench-Verified-eval150-turn24to12-8ckpts-2repeats-multicoordinator-w64-20260921
Coding eval150: 24→12 history and checkpoint screening
Primary goal: highest absolute accuracy. All selected outcomes and physical attempts are retained.
Code: https://github.com/ys-2020/miles/commit/65169383a9bbec29fc138c009d872032bc5ea084
Public evidence archive: https://huggingface.co/datasets/CharlieLLL/SWEbench-Verified-eval150-turn24to12-8ckpts-2repeats-multicoordinator-w64-20260921
Priority
Workers
Orchestrator
History
Independent full150 runs
A
8 candidates… See the full description on the dataset page: https://huggingface.co/datasets/CharlieLLL/SWEbench-Verified-eval150-turn24to12-8ckpts-2repeats-multicoordinator-w64-20260921.Physics-IQ-Verified
Physics-IQ Verified Dataset
This repository hosts the Physics-IQ Verified benchmark data for evaluating physical understanding in generative video models.
Physics-IQ Verified is derived from the original Physics-IQ benchmark dataset.
Original Physics-IQ
Paper: Do generative video models understand physical principles?
Repository: Code | Dataset in Google Cloud
Physics-IQ Verified (Recommended)
Paper: Physics-IQ Verified
Repository: Code | Dataset: Here in this repo :)
We… See the full description on the dataset page: https://huggingface.co/datasets/Anates-Labs-Research/Physics-IQ-Verified.atlas-31-strengthening-candidate-verification-under-rl
31. Strengthening candidate verification under reinforcement learning
1. Question and links
Read this first. The reading copy of this directory is t2ance/atlas-experiments under 31-strengthening-candidate-verification-under-rl/; the saved training steps and the per-token training arrays are on the Hugging Face repository t2ance/atlas-31-strengthening-candidate-verification-under-rl only.
How can reinforcement learning make the orchestrator's comparing and… See the full description on the dataset page: https://huggingface.co/datasets/t2ance/atlas-31-strengthening-candidate-verification-under-rl.SWE-bench_Not_Verified