Datasetpublic

Duke313/ClawBench-test

ClawBench Can AI Agents Complete Everyday Online Tasks? ClawBench evaluates AI agents on 153 everyday tasks (such as booking flights, ordering groceries, submitting job applications) across 144 live websites. We capture 5 layers of behavioral data (session replay, screenshots, HTTP traffic, agent reasoning traces, and browser actions), collect human ground-truth for every task, and score with an agentic evaluator that provides step-level traceable diagnostics. Paper… See the full description on the dataset page: https://huggingface.co/datasets/Duke313/ClawBench-test.

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes53downloads

Dataset Card

ClawBench

Can AI Agents Complete Everyday Online Tasks?

ClawBench evaluates AI agents on 153 everyday tasks (such as booking flights, ordering groceries, submitting job applications) across 144 live websites. We capture 5 layers of behavioral data (session replay, screenshots, HTTP traffic, agent reasoning traces, and browser actions), collect human ground-truth for every task, and score with an agentic evaluator that provides step-level traceable diagnostics.


Paper	arXiv:2604.08523
Website	claw-bench.com
Code	github.com/reacher-z/ClawBench
HF Papers	huggingface.co/papers/2604.08523
PyPI	clawbench-eval — `pip install clawbench-eval`

Dataset Structure

Columns

Column	Type	Description
`task_id`	int	Unique task identifier
`instruction`	string	Task prompt sent to the agent
`metaclass`	string	High-level category (21 categories)
`class`	string	Fine-grained sub-category
`platform`	string	Target platform (144 unique platforms)
`sites`	list[string]	Domains involved in the task
`eval_schema`	string (JSON)	Request interception configuration
`time_limit`	int	Maximum time in minutes
`extra_info`	string (JSON)	Paths to additional context files
`shared_info`	string	Path to shared user profile

Additional Files

shared/
  alex_green_personal_info.json   # Shared dummy user profile used across all tasks
extra_info/
  004/grocery_list.json           # Task-specific context (32 tasks have extra info)
  007/meal_plan.json
  043/pet_info.json
  ...

—`shared/alex_green_personal_info.json` — A comprehensive dummy user persona (Alex Green) including personal details, address, work history, education, financial information, and preferences. All tasks share this identity.
—`extra_info/` — Task-specific supplementary files referenced by the extra_info column. 32 of 153 tasks include additional context such as grocery lists, job links, meeting details, etc.

eval_schema

The eval_schema field configures the request interceptor — a mechanism that blocks the final HTTP request matching the specified URL pattern and method, preventing irreversible actions (checkout, form submission, etc.) from reaching the server. This allows safe evaluation on live websites.

json

{
  "url_pattern": "taskrabbit\\.(com|ca)/(api/v\\d+/jobs|book/\\d+/confirm)",
  "method": "POST"
}

Task Categories (metaclass)

Category	Tasks	Example Platforms
daily-life	21	Uber Eats, Instacart, Zillow
entertainment-hobbies	15	Goodreads, Eventbrite, Fandango
creation-init	13	ClickUp, Typeform, Ghost
office-secretary-tasks	9	Trello, Calendly, Purelymail
rating-voting	10	TripAdvisor, Glassdoor, Yelp
education-learning	9	Coursera, LeetCode, Blinkist
travel	9	Google Flights, Hipcamp, Airbnb
beauty-personal-care	9	TaskRabbit, Booksy, Soko Glam
pet-animal-care	8	Rover, Petfinder, Chewy
job-search-hr	8	Indeed, Greenhouse, ZipRecruiter
academia-research	5	Zotero, Overleaf, Google Scholar
and 10 more...

Usage

python

from datasets import load_dataset

ds = load_dataset("NAIL-Group/ClawBench", split="test")
print(ds[0])

Citation

bibtex

@article{zhang2026clawbench,
  title={ClawBench: Can AI Agents Complete Everyday Online Tasks?},
  author={Yuxuan Zhang and Yubo Wang and Yipeng Zhu and Penghui Du and Junwen Miao and Xuan Lu and Wendong Xu and Yunzhuo Hao and Songcheng Cai and Xiaochen Wang and Huaisong Zhang and Xian Wu and Yi Lu and Minyi Lei and Kai Zou and Huifeng Yin and Ping Nie and Liang Chen and Dongfu Jiang and Wenhu Chen and Kelsey R. Allen},
  journal={arXiv preprint arXiv:2604.08523},
  year={2026}
}