CoolFace
Datasetpublic

claw-eval/Claw-Eval

Claw-Eval End-to-end transparent benchmark for AI agents acting in the real world. Paper | Leaderboard | Code Dataset Structure Splits Split Examples Description general 161 Core agent tasks across 24 categories (communication, finance, ops, productivity, etc.) multimodal 101 Multimodal agentic tasks requiring perception and creation (webpage generation, video QA, document extraction, etc.) multi_turn 38 Multi-turn conversational tasks… See the full description on the dataset page: https://huggingface.co/datasets/claw-eval/Claw-Eval.

sourceHugging Facemitupdated 5mo agoView on Hugging Face
32likes3.3kdownloads
Dataset Card

<div align="center">

<h1>Claw-Eval</h1>

<img src="assets/claw_eval.png" alt="Claw-Eval Logo" width="200">

![Tasks](#dataset-structure) ![Leaderboard](https://claw-eval.github.io) ![License](https://github.com/claw-eval/claw-eval/blob/main/LICENSE)

End-to-end transparent benchmark for AI agents acting in the real world.

Paper | Leaderboard | Code

</div>

Dataset Structure

Splits

SplitExamplesDescription
general161Core agent tasks across 24 categories (communication, finance, ops, productivity, etc.)
multimodal101Multimodal agentic tasks requiring perception and creation (webpage generation, video QA, document extraction, etc.)
multi_turn38Multi-turn conversational tasks where the agent interacts with a simulated user persona to clarify needs and provide advice

Fields

FieldTypeDescription
task_idstringUnique task identifier
querystringTask instruction / description
fixturelist[string]Fixture files required for the task (available in data/fixtures.tar.gz)
languagestringTask language (en or zh)
categorystringTask domain

Usage

python
from datasets import load_dataset

# Load all splits
dataset = load_dataset("claw-eval/Claw-Eval")

# Load a specific split
general = load_dataset("claw-eval/Claw-Eval", split="general")
multimodal = load_dataset("claw-eval/Claw-Eval", split="multimodal")
multi_turn = load_dataset("claw-eval/Claw-Eval", split="multi_turn")

# Inspect a sample
print(general[0])

Acknowledgements

Our test cases are built on the work of the community. We draw from and adapt tasks contributed by OpenClaw, PinchBench, OfficeQA, OneMillion-Bench, Finance Agent, and Terminal-Bench 2.0.

Citation

If you use Claw-Eval in your research, please cite:

bibtex
@article{ye2026claw,
  title={Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents},
  author={Ye, Bowen and Li, Rang and Yang, Qibin and Liu, Yuanxin and Yao, Linli and Lv, Hanglong and Xie, Zhihui and An, Chenxin and Li, Lei and Kong, Lingpeng and others},
  journal={arXiv preprint arXiv:2604.06132},
  year={2026}
}

Core Contributors

Bowen Ye(PKU), Rang Li (PKU), Qibin Yang (PKU), Zhihui Xie(HKU), Yuanxin Liu(PKU), Linli Yao(PKU), Hanglong Lyu(PKU), Lei Li(HKU, project lead)

Advisors:

Tong Yang (PKU), Zhifang Sui (PKU), Lingpeng Kong (HKU), Qi Liu (HKU)

Contribution

We welcome any kind of contribution. Let us know if you have any suggestions!

License

This dataset is released under the MIT License.