claw-eval
Claw-Eval
Claw-Eval
End-to-end transparent benchmark for AI agents acting in the real world.
Paper | Leaderboard | Code
Dataset Structure
Splits
Split
Examples
Description
general
161
Core agent tasks across 24 categories (communication, finance, ops, productivity, etc.)
multimodal
101
Multimodal agentic tasks requiring perception and creation (webpage generation, video QA, document extraction, etc.)
multi_turn
38
Multi-turn conversational tasks where the… See the full description on the dataset page: https://huggingface.co/datasets/claw-eval/Claw-Eval.Auto-ClawEval
Auto-ClawEval
Auto-generated agent evaluation benchmark with 1,040 tasks across 104 unique scenarios created by ClawEnvKit.
Statistics
Tasks
1,040
Categories
24
Mock services
20
Task types
API-based (77%) + file-dependent (23%)
Quick Start
# Download
huggingface-cli download AIcell/Auto-ClawEval --repo-type dataset --local-dir Auto-ClawEval
# Evaluate with ClawEnvKit (Docker harness)
bash run_harnesses.sh --harness claudecode… See the full description on the dataset page: https://huggingface.co/datasets/AIcell/Auto-ClawEval.Auto-ClawEval-mini
Auto-ClawEval-mini
Compact agent evaluation benchmark with 104 tasks created by ClawEnvKit.
Statistics
Tasks
104
Categories
24
Mock services
20
Task types
API-based (77%) + file-dependent (23%)
Quick Start
# Download
huggingface-cli download AIcell/Auto-ClawEval-mini --repo-type dataset --local-dir Auto-ClawEval-mini
# Evaluate with ClawEnvKit (Docker harness)
bash run_harnesses.sh --harness claudecode --dataset Auto-ClawEval-mini… See the full description on the dataset page: https://huggingface.co/datasets/AIcell/Auto-ClawEval-mini.claw-eval-live
Claw-Eval-Live
A live benchmark for workflow agents: 105 controlled tasks with fixtures,
mock services, sandboxed workspaces, task-specific graders, and recorded
execution evidence. The release is a time-stamped snapshot built from public
workflow-demand signals, and the signal-to-task pipeline is designed to be
rerun as demand and models evolve.
This dataset accompanies an anonymous submission to the NeurIPS 2026
Evaluations and Datasets Track.
Quick facts
105… See the full description on the dataset page: https://huggingface.co/datasets/claw-eval-live/claw-eval-live.Visual-ClawEvaleval_CLAW_cupThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": null,
"total_episodes": 2,
"total_frames": 1500,
"total_tasks": 1,
"total_videos": 2,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:2"},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/smanni/eval_CLAW_cup.