datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
benchmark-bcplusexplicit-edit-benchmark
Explicit Edit Benchmark
226 deterministic exact-edit tasks, run by different agents, harnesses, models and configurations. Every observation records what the harness did and whether the resulting files matched byte for byte.
Source code and benchmark runner: GitHub — Explicit Edit Benchmark
Open the interactive Explorer to compare agents, harnesses, models, versions, reasoning modes, correctness, recovery, time, cost and tokens.
Leaderboard by model route
Score v2… See the full description on the dataset page: https://huggingface.co/datasets/alexshpunt/explicit-edit-benchmark.benchmark-research
Benchmark Research
面向金融 Deep Research Agent 的评测数据集（仅数据）。
代码、builder、prompt 模板与评测脚本在独立代码仓库维护；本 Hub 仓库只发布标准化 ready seeds，支持按子集按需加载。
快速开始
from datasets import load_dataset
# 按题型子集加载（推荐）
a1 = load_dataset("sselaine27/benchmark-research", "a1")
c = load_dataset("sselaine27/benchmark-research", "c")
d = load_dataset("sselaine27/benchmark-research", "d")
print(a1["train"][0]["task_id"])
print(a1["train"][0]["prompt"][:200])
# 按时间层筛选
t2 = a1["train"].filter(lambda x:… See the full description on the dataset page: https://huggingface.co/datasets/sselaine27/benchmark-research.regx-benchmark
RegX
Cross-Domain Multi-View Point Cloud Registration Benchmark
RegX evaluates multi-view point cloud registration across scales spanning nine orders
of magnitude — nanometre-scale microscopy to kilometre-scale airborne maps — and sensors
never designed to be compared: clinical colonoscopes, RGB-D cameras, spinning and
solid-state LiDAR, terrestrial and airborne laser scanners.
Most registration benchmarks fix one sensor and one scale. RegX asks a narrower question
instead: does… See the full description on the dataset page: https://huggingface.co/datasets/YuePanEdward/regx-benchmark.RoadmapBench
RoadmapBench
A benchmark for evaluating AI coding agents on multi-target, long-horizon software development tasks derived from open-source project version upgrades.
Overview
RoadmapBench contains 115 tasks spanning 17 open-source repositories across 5 programming languages (Python, TypeScript, Go, Rust, C++). Each task requires an agent to implement multiple interdependent features that correspond to a real version upgrade of the target project.
Task Structure… See the full description on the dataset page: https://huggingface.co/datasets/benchmark-anon-2026/RoadmapBench.cvdp-benchmark-datasetImportant please see "Files and versions" above for full list of files in the CVDP dataset.
Please see LICENSE and NOTICE for licensing information. See CHANGELOG for changes.
This is the Comprehensive Verilog Design Problems (CVDP) benchmark dataset to use with the CVDP infrastructure on GitHub.
funes-handoff-recall-benchmark
handover-vs-recall
A long investigation bloats an agent session until each new turn costs more to carry the context than to
do the work. Switching to a fresh session avoids that — but the findings have to travel somehow, and the
ways of moving them differ in cost. This benchmark measures those ways, as cost per successful task,
on tasks that genuinely require the prior investigation:
arm
channel
A branch-only
switch, carry nothing — the fresh session re-derives the… See the full description on the dataset page: https://huggingface.co/datasets/dacorvo/funes-handoff-recall-benchmark.data-product-benchmark
DPDisc Dataset
Paper | Code
Dataset Description
This dataset provides a benchmark for automatic data product creation. The task is framed as follows: given a natural language data product request and a corpus of text and tables, the objective is to identify the relevant tables and text documents that should be included in the resulting data product which would useful to the given data product request. The benchmark brings together three variants: HybridQA, TAT-QA, and… See the full description on the dataset page: https://huggingface.co/datasets/ibm-research/data-product-benchmark.FinQAFinQA is a large-scale dataset with 2.8k financial reports for 8k Q&A pairs to study numerical reasoning with structured and unstructured evidence.
Usage
import datasets
# Download the dataset
queries = datasets.load_dataset("embedding-benchmark/FinQA", "queries")
documents = datasets.load_dataset("embedding-benchmark/FinQA", "corpus")
pair_labels = datasets.load_dataset("embedding-benchmark/FinQA", "default")
FinanceBenchThe FinanceBench dataset is derived from the PatronusAI/financebench-test dataset, containing only the PASS examples processed into a clean format for question-answering tasks in the financial domain. FinanceBench-rtl has been repurposed for retrieval.
Usage
import datasets
# Download the dataset
queries = datasets.load_dataset("embedding-benchmark/FinanceBench", "queries")
documents = datasets.load_dataset("embedding-benchmark/FinanceBench", "corpus")
pair_labels =… See the full description on the dataset page: https://huggingface.co/datasets/embedding-benchmark/FinanceBench.amazon-benchmark
Amazon query–bundle benchmark
Canonical, category-organized query and reference-positive data. Experiment traces should reference this repository by commit SHA, category, split, and candidate_id, rather than republishing the dataset.
Musical Instruments
Split
Examples
agent_dev
2,028
agent_hidden
1,960
Each record contains a query and 3–7 reference product IDs. These are observed reference positives, not exhaustive labels for all valid… See the full description on the dataset page: https://huggingface.co/datasets/iaouali/amazon-benchmark.ogd4all-benchmark
OGD4All Benchmark
This is a 199-question benchmark that was used to evaluate the overall performance of OGD4All and different configurations (LLM, orchestration, ...).
OGD4All is an LLM-based prototype system enabling an easy-to-use, transparent interaction with Geospatial Open Government Data through natural language.
Each question requires GIS, SQL and/or topological operations on zero, one, or multiple datasets in GPKG or CSV formats to be answered.
Tasks
The… See the full description on the dataset page: https://huggingface.co/datasets/michael7ma/ogd4all-benchmark.HC3FinanceThe HC3 dataset comprises tens of thousands of comparison responses from both human experts and ChatGPT across various domains, including open-domain, financial, medical, legal, and psychological areas. The data collection process involved sourcing publicly available question-answering datasets and wiki texts, ensuring that the human answers were either expert-provided or high-quality user responses, thereby minimizing mislabeling and enhancing the dataset's reliability.
Usage
import datasets… See the full description on the dataset page: https://huggingface.co/datasets/embedding-benchmark/HC3Finance.scene-mem-benchmark
scene-mem-benchmark
A benchmark for scene memory in embodied agents: an agent watches a mobile manipulator work
in a house for several minutes, then is asked to retrieve an object it has to remember — one
that was moved, dropped, or merely seen along the way — or (resume) to go back and finish the
job it was interrupted in, remembering how far it had got — or (routine) to put a new object away
where this household keeps that kind of thing, a rule it was never told and can only… See the full description on the dataset page: https://huggingface.co/datasets/Keh0t0/scene-mem-benchmark.hack-ignition-benchmark
hack-ignition benchmark — data, v0.1.6
Training trajectories of reinforcement-learning runs on exploitable graders, for studying and predicting when RL
comes to produce exploits. Each family is a set of GRPO runs over configurations of (start model, prompt,
training set, grader / reward structure, recipe), with one or more seeds per configuration. Every family stores
what its training logs contain — per-step exploit, task and reward rates, the item × step exploit record… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/hack-ignition-benchmark.ChatDoctor_HealthCareMagicThe ChatDoctor-HealthCareMagic-100k dataset comprises 112,000 real-world medical question-and-answer pairs, providing a substantial and diverse collection of authentic medical dialogues. There is a slight risk to this dataset since there are grammatical inconsistencies in many of the questions and answers, but this can potentially help separate strong healthcare retrieval models from weak ones.
Usage
import datasets
# Download the dataset
queries =… See the full description on the dataset page: https://huggingface.co/datasets/embedding-benchmark/ChatDoctor_HealthCareMagic.SWE-QA-Benchmark
SWE-QA Benchmark
A comprehensive benchmark dataset for Software Engineering Question Answering, containing 720 questions across 15 popular Python repositories.
Dataset Summary
Total Questions: 720
Repositories: 15
Format: JSONL (JSON Lines)
Fields: question, answer
Repository Coverage
Each repository contains 48 questions:
astropy
conan
django
flask
matplotlib
pylint
pytest
reflex
requests
scikit-learn
sphinx
sqlfluff
streamlink
sympy
xarray… See the full description on the dataset page: https://huggingface.co/datasets/swe-qa/SWE-QA-Benchmark.mcp-agent-trajectory-benchmark
MCP Agent Trajectory Benchmark
A benchmark dataset of 49 MCP (Model Context Protocol) agent trajectories (38 single-pass + 11 multi-conv) with complete tool-use traces in the ATIF v1.2 (Agent Trajectory Interchange Format) format. Each agent operates in a distinct business domain with custom tools, realistic user conversations, and full execution traces.
Designed for training and evaluating tool-use / function-calling capabilities of LLMs.
Overview
Item
Details… See the full description on the dataset page: https://huggingface.co/datasets/obaydata/mcp-agent-trajectory-benchmark.AusLaw-Citation-BenchmarkThis is the dataset proposed in the paper: Methods for Legal Citation Prediction in the Age of LLMs: An Australian Law Case Study.
last-translation-benchmark
Last Translation Benchmark
Abstract: For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases.
Standard benchmarks for machine translation evaluation are often either trivial (having few authentic mistakes) or unrealistic (overly synthetically contrived).
Furthermore, automatic translation metrics become less reliable and reward-hacked as models get stronger, and their outputs are… See the full description on the dataset page: https://huggingface.co/datasets/zouhar/last-translation-benchmark.big-finance-benchmark
BigFinanceBench Public Release
arXiv | Website | GitHub | Blog post
Finance answers are only useful when another analyst can audit how they were produced. BigFinanceBench evaluates that full workflow: agents must produce a numerical answer, and their traces are graded against point-weighted rubrics for source choice, period, accounting definition, assumptions, adjustments, and calculation.
This release contains a 50-question stratified subset of the 928-item BigFinanceBench… See the full description on the dataset page: https://huggingface.co/datasets/RogoAI/big-finance-benchmark.Act2Cap_benchmarkCollected data from GUI-Action-Narrator
FreshStack_mtebUsage
import datasets
# Download the dataset
queries = datasets.load_dataset("embedding-benchmark/MBPP", "queries")
documents = datasets.load_dataset("embedding-benchmark/MBPP", "corpus")
pair_labels = datasets.load_dataset("embedding-benchmark/MBPP", "default")
inference-benchmarkerAuto-Fill-Benchmark
Auto-Fill Benchmark
Benchmark for predicting missing cell values in real-world tables, introduced in
Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models
(PVLDB 19(11), 2026 — arXiv:2607.19847).
Each case is a real table in which exactly one cell is replaced by [MISSING], together with the ground-truth value.
Code: https://github.com/lyrain2001/auto-fill
Models: Auto-Fill-Qwen3-8B-Knowledge ·
Auto-Fill-Qwen3-8B-Reasoning ·… See the full description on the dataset page: https://huggingface.co/datasets/lyrain2001/Auto-Fill-Benchmark.toc_bench
TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models
TOC-Bench is a diagnostic benchmark for evaluating whether Video Large Language Models maintain object identity, state, persistence, and temporal relations throughout a video. It focuses on object-centric phenomena including occlusion, disappearance, reappearance, repeated events, event order, temporal location, duration, conditional state, and relative movement.
Anonymous-review notice. This… See the full description on the dataset page: https://huggingface.co/datasets/anonymous-video-benchmark/toc_bench.map-spatial-benchmark
Map-based Spatial Reasoning Benchmark
A multi-view map-based spatial reasoning benchmark. Each row is one multiple-choice
question instance over a registered map image; models must answer with a single option
letter. Four tasks (T1–T4), four base-map views, and controlled evidence conditions
(direct / query / oracle) and world perturbations (transform / world layers) allow
fine-grained analysis of spatial reasoning robustness.
Task overview
Task
Question… See the full description on the dataset page: https://huggingface.co/datasets/mapspatial/map-spatial-benchmark.MBPPThe MBPP dataset consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. As described in the paper, a subset of the data has been hand-verified by the dataset authors to ensure quality.
Usage
import datasets
# Download the dataset
queries =… See the full description on the dataset page: https://huggingface.co/datasets/embedding-benchmark/MBPP.Multi-turn_Long-context_Benchmark_for_LLMs
LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues
Arxiv: https://www.arxiv.org/abs/2507.13681
Huggingface: https://huggingface.co/papers/2507.13681
Introduction
LoopServe Multi-Turn Dialogue Benchmark is a comprehensive evaluation dataset comprising multiple diverse datasets designed to assess large language model performance in realistic conversational scenarios.
Unlike traditional benchmarks that place queries only at the end… See the full description on the dataset page: https://huggingface.co/datasets/TreeAILab/Multi-turn_Long-context_Benchmark_for_LLMs.HumanEvalThe HumanEval dataset released by OpenAI includes 164 programming problems with a handwritten function signature, docstring, body, and several unit tests for each problem. The dataset was handcrafted by engineers and researchers at OpenAI.
Usage
import datasets
# Download the dataset
queries = datasets.load_dataset("embedding-benchmark/HumanEval", "queries")
documents = datasets.load_dataset("embedding-benchmark/HumanEval", "corpus")
pair_labels =… See the full description on the dataset page: https://huggingface.co/datasets/embedding-benchmark/HumanEval.