datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Mind2Web
Dataset Card for Dataset Name
Dataset Summary
Mind2Web is a dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website. Existing datasets for web agents either use simulated websites or only cover a limited set of websites and tasks, thus not suitable for generalist web agents. With over 2,000 open-ended tasks collected from 137 websites spanning 31 domains and crowdsourced action… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/Mind2Web.SuperMemory-VQA
SuperMemoryVQA
SuperMemory-VQA is an egocentric visual question answering benchmark for
evaluating long-horizon memory in augmented reality assistant settings. The
dataset is designed around practical questions a person might ask a wearable
memory assistant, such as where an object was left, what someone said earlier,
whether a planned step was completed, or what happened next in a longer event.
The benchmark contains 4,853 human-verified question-answer pairs grounded in
52.9… See the full description on the dataset page: https://huggingface.co/datasets/OSU-AIoT-MLSys-Lab/SuperMemory-VQA.SkillHarm
SkillHarm
Lifecycle-Aware Skill-Based Attacks via Automated Construction
📄 Paper · 🌐 Project Page · 💻 GitHub · 🤗 Data
Agent skills occupy a privileged position in the agent workflow — agents are expected to implicitly follow and execute them — which makes third-party skills a vulnerable supply-chain attack surface. SkillHarm is a benchmark of skill-based attacks across the skill-use lifecycle, paired with a systematic taxonomy of 12 skill-relevant risks. Every attack is… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/SkillHarm.AttributionBench
Dataset Card for AttributionBench
Github repository: [Github]
Paper: AttributionBench: How Hard is Automatic Attribution Evaluation?
Point of Contact: Yifei Li
Dataset Overview
We constructed this dataset from multiple existing data sources in a unified format, in order to create a unified and diverse testbed for evaluating advanced attribution evaluation systems. The dataset contains both in-domain training set and id-domain and out-of-domain test set.
Usage… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/AttributionBench.Online-Mind2Web
Blog |
Paper |
Code |
Leaderboard
Online-Mind2Web
Online-Mind2Web is the online version of Mind2Web, a more diverse and user-centric dataset includes 300 high-quality tasks from 136 popular websites across various domains. The dataset covers a diverse set of user tasks, such as clothing, food, housing, and transportation, to evaluate web agents' performance in a real-world online environment.
News
[11/03/2025] We’ve updated 36 tasks that are… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/Online-Mind2Web.early-experience
Early Experience — Reproduction Data
Supervised fine-tuning data for reproducing Agent Learning via Early Experience across 8 agent environments. Each environment provides data for three training paradigms:
IL — Imitation Learning: expert
SR — Self-Reflection: expert + reflection
IWM — Implicit World Modeling: iwm (world model) → expert
Code: OSU-NLP-Group/EarlyExperience
Usage
from datasets import load_dataset
# load_dataset("osunlp/early-experience"… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/early-experience.AutoElicit-Exec
AutoElicit-Exec Dataset
Project Page | Paper | GitHub
AutoElicit-Exec is a human-verified dataset of 132 execution trajectories exhibiting unintended behaviors from typical benign execution. All trajectories are elicited from frontier CUAs (i.e., Claude 4.5 Haiku and Claude 4.5 Opus) using AutoElicit, which perturbs benign instructions from OSWorld to increase the likelihood of unintended harm while keeping instructions realistic and benign. This dataset is designed to provide… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/AutoElicit-Exec.TACO-Cobalt
TACO-Cobalt
Paper | GitHub
TACO-Cobalt is a cleaned version of that TACO code generation dataset crawled from programming competition websites.
Due to noise in the crawled data, we start from the TACO-verified subset and apply further cleaning to obtain 6,103 valid tasks, each with at least 8 test cases.
For each task, we sort the test cases’ difficulty by the possibility of Qwen2.5-Coder-7B-Instruct passing them out of 16 attempts.
We select the four easiest test cases as the… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/TACO-Cobalt.AgentCL
AgentCL: Evaluation Framework for Continual Learning in Agents
Datasets
agentboard_babyai, agentboard_scienceworld, and mmlu_pro are included as ready-to-use subsets. They are direct subsets from existing public datasets, with no modifications to the original data.
Our constructed codeeval-pro and browsecomp_plus streams are provided.
The metadata in source datasets only indicates the authorship for the source datasets, and does not imply the authorship of this… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/AgentCL.AutoSDT-5K
AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists
AutoSDT-5K is an automatically constructed dataset of 5,404 coding tasks for data-driven discovery that covers four scientific disciplines and 756 unique Python packages. Expert feedback on a subset of 256 tasks shows the quality of AutoSDT-5K: 93% of the collected tasks are ecologically valid, and 92.2% of the synthesized programs are functionally correct. To the best of our knowledge, AutoSDT-5K is the only… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/AutoSDT-5K.AutoElicit-Bench
AutoElicit-Bench
Project Page | Paper | GitHub
Overview
AutoElicit-Bench is a benchmark of 117 human-verified perturbations for evaluating the robustness of Computer-Use Agents (CUAs) against unintended behaviors. These perturbations were successfully elicited from Claude 4.5 Haiku (50) and Claude 4.5 Opus (67) using the AutoElicit framework.
Each perturbation is a minimally modified, realistic instruction derived from OSWorld tasks that can trigger unsafe behaviors… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/AutoElicit-Bench.TACO-Cobalt-PTB
TACO-Cobalt-PTB
Paper | GitHub
TACO-Cobalt is the perturbed version of the validation set in TACO-Cobalt to analyze in-context reward hacking behaviors of LLMs in code generation contexts.
For each coding problem, we randomly select two public tests (x_1, y_1) and (x_2, y_2) with distinct outputs (y_1 != y_2).
Then, we exchange their expected outputs and turn them into two perturbed tests (x_1, y_2) and (x_2, y_1), which are impossible for any correct program to pass.
If all public… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/TACO-Cobalt-PTB.D3-Gym-Trajectories
D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery
D3-Gym is the first automatically constructed dataset of verifiable environments for Data-Driven Discovery. It contains 565 tasks derived from 239 real-world multi-disciplinary scientific repositories.
The present dataset contains all training trajectories used in our paper, with each split representing the trajectories sampled from a model among the Qwen3 family.
Citation
If you find… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/D3-Gym-Trajectories.KBQA-AgentIntroduction
In traditional knowledge base question answering (KBQA) methods, semantic parsing plays a crucial role. It requires a semantic parser to be extensively trained on a vast dataset of labeled examples, typically consisting of question-answer or question-program pairs.
However, the rise of LLMs has shifted this paradigm. LLMs excel in learning from few (or even zero) in-context examples. They utilize natural language as a general vehicle of thought, enabling them to actively navigate… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/KBQA-Agent.AutoElicit-Seed
AutoElicit-Seed Dataset
Project Page | Paper | GitHub
AutoElicit-Seed is a dataset of 361 seed perturbations across 66 benign tasks in the OSWorld domain that are used to guide elicitation efforts with AutoElicit. These seeds include perturbations to benign OSWorld instructions that are designed to increase the likelihood of eliciting unsafe unintended behaviors from frontier Computer-Use Agents while remaining realistic and benign, allowing us to scalably surface safety risks from… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/AutoElicit-Seed.