CoolFace
Datasetpublic

wanlilll/WeaveBench

WeaveBench A long-horizon, real-world benchmark for computer-use agents with hybrid GUI + CLI + code interfaces. 🎊 Accepted to EMNLP 2026 Main Conference — see you in Budapest! 📄 Paper: arXiv:2606.09426 💻 Code: github.com/weavebench/WeaveBench 🌐 Website: weavebench.github.io 114 long-horizon, real-world tasks across 8 work domains, where every task requires the agent to interleave GUI clicks with shell/code in one trajectory. Scored by a trajectory-aware Agent-as-Judge… See the full description on the dataset page: https://huggingface.co/datasets/wanlilll/WeaveBench.

sourceHugging Facemitupdated 1mo agoView on Hugging Face
8likes2.5kdownloads
Dataset Card

WeaveBench

A long-horizon, real-world benchmark for computer-use agents with hybrid GUI + CLI + code interfaces.

🎊 Accepted to [EMNLP 2026](https://2026.emnlp.org/) Main Conference — see you in Budapest!

📄 Paper: arXiv:2606.09426 💻 Code: github.com/weavebench/WeaveBench 🌐 Website: weavebench.github.io

114 long-horizon, real-world tasks across 8 work domains, where every task requires the agent to interleave GUI clicks with shell/code in one trajectory. Scored by a trajectory-aware Agent-as-Judge that reads chat trace + deliverables and emits per-clause evidence — much harder to spoof than file-existence checks. Best observed pairing in the paper: Claude Opus 4.7 + Claude Code at 41.2 % PassRate — far from saturation.

This HF repository hosts everything needed to run the benchmark end-to-end:

  1. 1.`tasks/` — 114 paper-final tasks + workspace assets (~207 MB)
  2. 2.`runtime_assets/` — pre-built bootstrap tarballs for the 4 in-VM agent harnesses (~852 MB)
  3. 3.`vm/Ubuntu.qcow2` — the paper-canonical Ubuntu VM image (28.46 GB)
  4. 4.`judge/judge_template.tar.gz` — host-side OpenClaw judge profile template (~7 KB)
🇨🇳 Users in China: export HF_ENDPOINT=https://hf-mirror.com — all weavebench-download-* commands honor it automatically.

1. Layout

WeaveBench/
├── tasks/                                ← 207 MB
│   ├── DAV/  DES/  DOC/  DSK/  GAM/  OPS/  SPA/  WEB/    ← 8 domains
│   │   └── <DOMAIN>_task_<NN>_<slug>.md  ← 114 task .md files
│   └── workspace/<DOMAIN>/<task_dir>/    ← 951 supporting files (gt/, exec/, …)
│
├── runtime_assets/                       ← 852 MB total
│   ├── openclaw.tar.gz                   514 MB   (reference harness)
│   ├── codex.tar.gz                      131 MB   (OpenAI Codex CLI)
│   ├── claudecode.tar.gz                  72 MB   (Anthropic Claude Code)
│   ├── hermes.tar.gz                     127 MB   (Nous Research Hermes)
│   └── hermes_mcp_wheels.tar.gz            9 MB   (offline mcp wheels for Hermes)
│
├── vm/
│   └── Ubuntu.qcow2                      28.46 GB (v3_eyeson_apps, paper-canonical)
│
└── judge/
    └── judge_template.tar.gz             ~7 KB    (OpenClaw judge profile)

Total: ~29.5 GB. See the README §Resource budget on the code repo for what to allocate.

2. One-command download + run

bash
git clone https://github.com/weavebench/WeaveBench.git && cd WeaveBench

export OPENROUTER_API_KEY=sk-or-v1-...
# Optional: export HF_ENDPOINT=https://hf-mirror.com   # if in China

# bash scripts/setup.sh does all four downloads + npm install -g openclaw + judge bootstrap
bash scripts/setup.sh

# Smoke test (default model: openai/gpt-5.5)
weavebench-run \
    --harness openclaw \
    --model openai/gpt-5.5 \
    --tasks_root ./cache/tasks \
    --domains WEB --task_filter task_1_ \
    --result_dir ./results/smoke

Or fetch individual assets:

bash
weavebench-download-dataset --dest ./cache         # tasks/             207 MB
weavebench-download-assets  --dest ./cache         # runtime_assets/    852 MB total
weavebench-download-vm      --dest ./cache         # vm/Ubuntu.qcow2    28.46 GB
weavebench-download-judge                          # judge/ → ~/judge_agent_test/

3. Try it in 30 seconds (no VM, no API key)

If you just want to see what score.json from the Agent-as-Judge looks like:

bash
git clone https://github.com/weavebench/WeaveBench.git && cd WeaveBench
pip install -e .
weavebench-demo

This prints a real paper-canonical Claude Opus 4.7 rollout's score with per-artifact / per-clause judge evidence in <1 second — no network calls, no tokens.

4. Task .md schema

Each task file has these sections in order:

markdown
# <Human title>

## Goal             — short user request (what the agent reads as `instruction`)
## Setup            — preconditions assumed true in the VM
## Warmup           — bash commands the orchestrator runs before the agent starts
## Expected Output  — files the agent must produce in /tmp_workspace/results/
## Grader           — Python `def grade(workspace_path, transcript) -> dict`

The embedded grade(...) function returns {"score": float ∈ [0, 1], "scores": {sub_rubric: float, ...}, "msg": "..."} — but in the trajectory-aware judge pipeline (the only scoring path in weavebench-run), this is documentation only. Scoring is done by a host-side OpenClaw judge (`weavebench/eval/agent_judge`) that reads chat trace + deliverables and emits per-clause evidence-based scores. See docs/AGENT_JUDGE.md.

5. Pinned revision for paper reproduction

OpenRouter aliases drift over time. To reproduce paper numbers, pin both the dataset SHA and the model snapshot ids — see `docs/REPRODUCE.md` in the code repo for the canonical SHA, per-table model ids, and per-table commands.

bash
export WEAVEBENCH_DATASET_REVISION=<full sha from docs/REPRODUCE.md>

6. Citation

bibtex
@article{li2026weavebench,
  title         = {WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces},
  author        = {Li, Wanli and Zhou, Bowen and Yu, Yunyao and Xu, Zhou and Yang, Yifan and Li, Dongsheng and Shan, Caihua},
  year          = {2026},
  eprint        = {2606.09426},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2606.09426},
}

7. License

  • —Tasks: MIT.
  • —Runtime tarballs: each tarball repackages third-party software (Codex CLI is Apache-2.0; Claude Code, Hermes, OpenClaw retain their upstream terms). See NOTICE in the code repo for full attribution.
  • —VM image: derived from OSWorld upstream Ubuntu image (MIT), patched to bake in app snapshots for paper reproducibility — full provenance in docs/REPRODUCE.md.