internlm/WildClawBench

<h1 align="center">WildClawBench</h1>

<p align="center"> <img src="https://huggingface.co/datasets/internlm/WildClawBench/resolve/main/assets/lobster_battle.png" alt="WildClawBench Lobster" width="480"> </p>

<div align="center">

![Tasks]() ![Harnesses]() ![Models]()  <br>      <br> [](https://huggingface.co/datasets/internlm/WildClawBench-Harbor) [](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories)

</div>

Hard, practical, end-to-end evaluation for AI agents — in the wild.

WildClawBench is an agent benchmark that tests what actually matters: can an AI agent do real work, end-to-end, without hand-holding?

We drop agents into a live OpenClaw environment — the same open-source personal AI assistant that real users rely on daily — and throw 60 original tasks at them: clipping goal highlights from a football match, negotiating meeting times over multi-round emails, hunting down contradictions in search results, writing inference scripts for undocumented codebases, catching privacy leaks before they happen. Useful things. Hard things.

In the technical report snapshot, the strongest frontier model topped out at 62.2% overall; the latest audited OpenClaw runs have since raised the best score to 67.2%. Most models still land well below that. That makes scores mean something.

Why WildClawBench?

Most agent benchmarks test isolated capabilities — calling a function, parsing JSON, following a single instruction. WildClawBench tests the full picture:

What Sets Us Apart

• —Real environment, not mocks. Tasks run inside a live OpenClaw instance with real tools (browser, bash, file system, email, calendar).
• —60 original tasks, built by hand. Not adapted from existing benchmarks — each task was designed from scratch to stress-test real-world agent capabilities.
• —Four agent harnesses, one task suite. OpenClaw, Claude Code, Codex CLI, and Hermes Agent all execute the same 60 tasks under the same grading. This separates model capability from harness scaffolding — you can see how much an agent's score depends on its surrounding tools versus the underlying LLM.
• —Reproducible & isolated. Each task runs in its own Docker container. Same image, same data, same grading code. Ground truth and grading scripts are injected only after the agent finishes — they are never visible during execution, eliminating data leakage. Scores are reproducible across machines.

The WildClawBench Family

WildClawBench ships as three Hugging Face repositories — pick the one that matches how you want to use the benchmark:

News

• —2026-08 Meta's [Muse Glimmer release](https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model) reports WildClawBench evaluation scores. Thanks for the recognition!
• —2026-08 Released [WildClawBench-Harbor](https://huggingface.co/datasets/internlm/WildClawBench-Harbor) — the full 60-task suite in Harbor format — and [WildClawBench-Trajectories](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories) — complete agent trajectories from our frontier-model evaluations, browsable in the HF Agent Trace Viewer and continuously updated as new models are evaluated.
• —2026-07 We expanded the OpenClaw leaderboard with evaluations of the latest frontier models, including GPT-5.6 Sol, Claude Fable 5, Kimi K3 and etc.
• —2026-06 ByteDance Seed's [Seed2.1 release](https://seed.bytedance.com/en/blog/seed2-1-officially-released-advancing-ai-productivity) includes WildClawBench in its agent evaluations. Thanks for the recognition!
• —2026-05 We released a new version with four agent harnesses — OpenClaw, Claude Code, Codex CLI, and Hermes Agent — so the same 60-task suite can be evaluated under multiple scaffolds.
• —2026-05 We published a [technical report PDF](https://github.com/internlm/WildClawBench/blob/main/WildClawBench_report.pdf).
• —2026-05 Tencent’s [Hunyuan3 Preview](https://hunyuan.tencent.com/research/hy3) page reports WildClawBench evaluation scores. Thanks for the recognition!

Leaderboard

WildClawBench reports two complementary leaderboards:

1. 1.Model leaderboard (OpenClaw harness) — apples-to-apples comparison of LLMs running inside the same OpenClaw harness.
2. 2.Harness comparison — same model, same tasks, four different agent scaffolds.

Full interactive leaderboard at internlm.github.io/WildClawBench.

Model leaderboard (OpenClaw harness)

Overall score follows the weighted Multimodal / Pure-text breakdown in that table. Total time and total cost are the paper’s Overall per-task averages (minutes / USD) multiplied by 60 for the full 60-task suite.

Claude Opus 4.8 cost uses the dynamic base-tier rates for this evaluation: $5/M input, $25/M output, $0.5/M cache read, and $6.25/M cache write. Kimi K2.7 Code cost uses the published rates for this evaluation: $6.5/M input, $27/M output, and $1.3/M cached input.

Harness comparison

Same 60 tasks, same grading, four different agent scaffolds. Time and cost are per-task averages; score is in %. Time is in minutes per task, cost in USD per task. Bold = best harness for that model.

Tasks

60 tasks across 6 categories, spanning English and Chinese:

To create new tasks, see the annotated template at `tasks/task0_template.md`.

Quick Start

This Hugging Face repository hosts the benchmark's large Docker images and task data. The evaluation code is maintained in the GitHub repository. Clone the code repository first, then download the data below into its root directory. Prefer a standard runner? The suite is also available in Harbor format — skip the setup below and jump to Run with Harbor.

git clone https://github.com/internlm/WildClawBench.git
cd WildClawBench

Install Docker

<details> <summary>macOS</summary>

brew install --cask docker

After installation, launch Docker Desktop from Applications or run:

open -a Docker

</details>

<details> <summary>Ubuntu</summary>

# Install dependencies
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

# Add Docker's official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add apt repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

# Allow current user to run Docker without sudo
sudo usermod -aG docker $USER
newgrp docker

</details>

Download Images

WildClawBench ships four Docker images, one per harness. They are all hosted on HuggingFace. Pick the one(s) that match the harness you want to evaluate:

pip install -U "huggingface_hub[cli]"

# Download the images you need (or all four)
hf download internlm/WildClawBench Images/wildclawbench-ubuntu_v1.3.tar                    --repo-type dataset --local-dir .
hf download internlm/WildClawBench Images/wildclawbench-claudecode-ubuntu_v0.2-patched.tar --repo-type dataset --local-dir .
hf download internlm/WildClawBench Images/wildclawbench-codex-ubuntu_v0.0.tar              --repo-type dataset --local-dir .
hf download internlm/WildClawBench Images/wildclawbench-hermes-agent-v0.5.tar.gz           --repo-type dataset --local-dir .

Then load each image into Docker:

docker load -i Images/wildclawbench-ubuntu_v1.3.tar
docker load -i Images/wildclawbench-claudecode-ubuntu_v0.2-patched.tar
docker load -i Images/wildclawbench-codex-ubuntu_v0.0.tar
docker load -i Images/wildclawbench-hermes-agent-v0.5.tar.gz

Download Task Data

Download the task data from HuggingFace:

hf download internlm/WildClawBench workspace --repo-type dataset --local-dir .

Prepare Data

Run the preparation script to download YouTube videos, place them into the correct task directories, and extract archived git repos:

bash script/prepare.sh

The script will:

• —Download 3 YouTube videos (football match, lecture, product launch event)
• —Extract the first half of the football match and discard the full video
• —Rename and copy videos to the directories that need them
• —Extract dot_git.tar.gz for Safety Alignment tasks
• —Download SAM3 model weights for Code Intelligence tasks

Prerequisites: yt-dlp, ffmpeg, gdown.

Note: YouTube downloads may require authentication. If you encounter a "Sign in to confirm you're not a bot" error, try one of the following: - Get cookies.txt locally. - Use --cookies-from-browser (e.g., --cookies-from-browser chrome) - Install Deno as a JS engine, which some users have reported resolves the issue

Run

Set your API keys in the .env file:

OPENROUTER_API_KEY=your_api_key_here
BRAVE_API_KEY=your_brave_key_here  # required for search tasks

• —OpenRouter API Key — Any model available on OpenRouter is supported. The default model is defined in the .env file as DEFAULT_MODEL=openrouter/stepfun/step-3.5-flash:free — replace it with any model you want to evaluate.
• —Brave Search API Key — Required for Search & Retrieval tasks. Get one (with free monthly credits) at brave.com/search/api.
• —Judge model (optional) — JUDGE_MODEL controls the LLM used by judge-based grading metrics. Defaults to openai/gpt-5.4.

Then run one of the four harnesses:

bash script/run.sh openclaw     --category all --parallel 4 --model openrouter/openai/gpt-5.5
bash script/run.sh claudecode   --category all --parallel 4 --model openai/gpt-5.5
bash script/run.sh codex        --category all --parallel 4 --model openrouter/openai/gpt-5.5
bash script/run.sh hermesagent  --category all --parallel 4 --model openai/gpt-5.5

Single-task runs are also supported:

bash script/run.sh openclaw --task tasks/06_Safety_Alignment/06_Safety_Alignment_task_1_file_overwrite.md \
                            --model openrouter/openai/gpt-5.5

Model-name conventions differ per harness: - OpenClaw / Codex expect openrouter/<provider>/<model> (since they hit OpenRouter directly). - Claude Code / Hermes Agent expect <provider>/<model> (the openrouter/ prefix is added internally).

Using a Custom Model Endpoint (Without OpenRouter)

This option currently applies to the OpenClaw harness only. If you prefer to use your own API endpoint instead of OpenRouter, you can provide a JSON file and WildClawBench will inject it into ~/.openclaw/openclaw.json before each task starts.

⚠️ Important: Some task prompts and evaluation scripts currently have OpenRouter explicitly mentioned or hardcoded (e.g., https://openrouter.ai/api/v1). If you bypass OpenRouter, you will need to adjust these references in the respective files manually.

1. Fill in `my_api.json` (or provide your own JSON file with the same format):

{
  "providers": {
    "my-openai-proxy": {
      "baseUrl": "http://host.docker.internal:8000/v1",
      "apiKey": "${MY_PROXY_API_KEY}",
      "api": "openai-completions",
      "models": [
        {
          "id": "my-model",
          "name": "My Model"
        }
      ]
    }
  }
}

This file is the value written into openclaw.json["models"], so it should contain the models object itself, not the full openclaw.json. If you use ${MY_PROXY_API_KEY}, WildClawBench will replace it on the host before the config is copied into the container, so MY_PROXY_API_KEY must be set in .env. WildClawBench always replaces the existing top-level models field with the JSON you provide.

2. Set your model name and required API key in `.env`:

MY_PROXY_API_KEY=your_api_key_here

3. Run the benchmark with the models config file:

python3 eval/run_batch.py --category 01_Productivity_Flow --models-config my_api.json --model my-openai-proxy/my-model

<details> <summary>Common provider examples</summary>

OpenAI-compatible proxy:

{
  "providers": {
    "proxy": {
      "baseUrl": "http://host.docker.internal:8000/v1",
      "models": [
        {
          "id": "gpt-4o",
          "name": "GPT-4o"
        }
      ]
    }
  }
}

Local vLLM or LM Studio:

{
  "providers": {
    "local-openai": {
      "baseUrl": "http://host.docker.internal:1234/v1",
      "models": [
        {
          "id": "qwen2.5-coder-32b-instruct",
          "name": "Qwen2.5 Coder 32B Instruct"
        }
      ]
    }
  }
}

Provider with explicit API mode and env var key:

{
  "providers": {
    "custom-gateway": {
      "baseUrl": "http://host.docker.internal:9000/v1",
      "apiKey": "${MY_PROXY_API_KEY}",
      "api": "openai-completions",
      "models": [
        {
          "id": "my-reasoning-model",
          "name": "My Reasoning Model"
        }
      ]
    }
  }
}

</details>

Run with Harbor

The full suite is also published in the Harbor task format at [internlm/WildClawBench-Harbor](https://huggingface.co/datasets/internlm/WildClawBench-Harbor). Each of the 60 tasks is a self-contained Harbor task directory (task.toml / instruction.md / environment/ / tests/), with task content and grading logic identical to this repository. This is the easiest way to evaluate agents that Harbor already supports (Claude Code, OpenHands, Codex CLI, custom agents, ...) — no benchmark-specific pipeline needed.

uv tool install harbor   # or: pip install harbor

# Task suite
hf download internlm/WildClawBench-Harbor --repo-type dataset --local-dir ./WildClawBench-Harbor

# Docker image (same OpenClaw image as this repo)
hf download internlm/WildClawBench Images/wildclawbench-ubuntu_v1.3.tar --repo-type dataset --local-dir .
docker load -i Images/wildclawbench-ubuntu_v1.3.tar

# Run the full benchmark (or point -p at a single task directory)
harbor run -p ./WildClawBench-Harbor -a claude-code -m anthropic/claude-opus-4-1 --n-concurrent 4

See the WildClawBench-Harbor card for the task layout, environment details, and scoring.

Check the Results

After the run completes, a per-category summary and a global summary (output/summary_all.json) are generated automatically. Each metric is scored from 0.00 to 1.00.

Per-task results are saved under output/<harness>/<category>/<task_id>/<model_timestamp_runid>/:

output/<harness>/<category>/<task_id>/<model_timestamp_runid>/
├── score.json          # per-metric scores
├── usage.json          # token counts, cost, elapsed time
├── agent.log           # agent execution log
├── chat.jsonl          # full conversation trace (OpenClaw)
├── claude_code_log/    # Claude Code session log (Claude Code)
├── codex_sessions/     # Codex session JSONLs (Codex)
├── gateway.log         # gateway log (OpenClaw)
└── task_output/        # files produced by the agent

The subdirectory name is <short_model>_<timestamp>_<runid>, where short_model is the last segment of the model path (e.g. claude-sonnet-4.6 from openrouter/anthropic/claude-sonnet-4.6) and runid is a 6-char random hex string, so parallel or repeated runs never collide.

Agent Trajectories

For independent verification, side-by-side comparison, and trace-level analysis, we release [internlm/WildClawBench-Trajectories](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories): complete OpenClaw trajectories covering the full 60-task suite for each evaluated model — including recent frontier models such as GPT-5.6 Sol, Claude Fable 5, Claude Opus 4.8, Kimi K3, and more. The collection is continuously updated as new models join the leaderboard; see the dataset card for the current roster. The same data is provided in three forms:

• —`train.parquet` — one row per (task, model) with the full message sequence as a JSON array; loads directly with load_dataset("internlm/WildClawBench-Trajectories") and renders in the HF Dataset Viewer (inline images replaced by hash placeholders to keep rows small).
• —`sessions/<model>/<task_id>.jsonl` — per-session trace files for the HF Agent Trace Viewer: open any file, select the Trace tab, and step through reasoning blocks, tool calls, tool results, and token usage. These preserve the original inline image data.
• —*`output_.tar.gz`** — the raw per-task evaluation outputs (scores, usage, logs, agent-produced files) exactly as generated by the pipeline above.

Earlier evaluation details remain available on Google Drive:

• —overallresults.json: [Overall Results](https://drive.google.com/file/d/1EI1ABNLwEaiguzUU7f0RuEk5KFIMLUu/view?usp=drive_link)
• —overalldashboard.html: [Performance Dashboard](https://drive.google.com/file/d/1B7nStKfXeyATBM3lIv858M9FaH6QBPWU/view?usp=drivelink)
• —gemini 3.1 Pro Details: Gemini 3.1 Pro
• —GPT 5.4 Details: GPT 5.4
• —Kimi K2.5 Details: Kimi K2.5
• —MiniMax M2.7 Details: MiniMax M2.7
• —Claude Opus 4.6 Details: Claude 4.6 Opus

Personal OpenClaw Evaluation

"Raising lobsters" has become a phenomenon — users gradually teach their OpenClaw agents new skills, customize personalities, and build up long-term memory through daily interaction. A natural question follows: whose lobster is better? Beyond bragging rights, there is real value in understanding which skill combinations, persona designs, and memory strategies actually improve agent performance on a given model. That's why we created the Personal OpenClaw Leaderboard. Submit your lobster's results and see how it stacks up!

python eval/run_batch.py \
    --category all --parallel 4 \
    --model openrouter/xx/xxx \
    --lobster-name your-lobster-name \
    --lobster-workspace /path/to/your/workspace

• —--lobster-name — identifier, used in the output directory.
• —--lobster-workspace — path to your OpenClaw workspace (containing SOUL.md, USER.md, MEMORY.md, skills/, etc.).
• —--lobster-env — (optional) comma-separated env var names for skills that need API keys (e.g. GEMINI_API_KEY,FIRECRAWL_API_KEY). Add the actual values to .env.

After the run completes, send the following to wildclawbench@proton.me:

1. 1.Your output/summary_all_<lobster-name>_<model>.json
2. 2.(Optional) A brief description of how you trained your OpenClaw (e.g. key skills, custom SOUL.md, memory strategies).

We will update the leaderboard periodically.

Citation

If you use WildClawBench in your research, please cite it as:

@article{ding2026wildclawbench,
  title={WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation},
  author={Ding, Shuangrui and Dai, Xuanlang and Xing, Long and Ding, Shengyuan and Liu, Ziyu and JingYi, Yang and Yang, Penghui and Zhang, Zhixiong and Wei, Xilin and Fang, Xinyu and others},
  journal={arXiv preprint arXiv:2605.10912},
  year={2026}
}

For machine-readable citation metadata, see `CITATION.cff`. GitHub will use this file to populate the repository's "Cite this repository" panel.

Contributors

Shuangrui Ding\ (Project Lead), [Xuanlang Dai](https://github.com/LennoxDai)\, Long Xing\*, Shengyuan Ding, Ziyu Liu, Jingyi Yang, Penghui Yang, Zhixiong Zhang, Xilin Wei, Xinyu Fang

Advisors: Yubo Ma, Haodong Duan, Jing Shao, Jiaqi Wang, Dahua Lin, Kai Chen, Yuhang Zang

Acknowledgements

WildClawBench builds on top of the excellent open-source agent ecosystem. We gratefully acknowledge the following projects:

• —[OpenClaw](https://github.com/openclaw/openclaw)
• —[Claw-Eval](https://github.com/claw-eval/claw-eval)
• —[PinchBench](https://github.com/pinchbench/skill)
• —[Hermes-Agent](https://github.com/nousresearch/hermes-agent)

Cleanup

If a run is interrupted (e.g. Ctrl+C, terminal closed), some Docker containers may be left behind. To remove all WildClawBench containers when no tasks are running:

for img in \
    wildclawbench-ubuntu:v1.3 \
    wildclawbench-claudecode-ubuntu:v0.2 \
    wildclawbench-codex-ubuntu:v0.0 \
    wildclawbench-hermes-agent:v0.5; do
  docker ps -a --filter "ancestor=$img" -q | xargs -r docker rm -f
done

To preview which containers would be removed (dry run), drop the docker rm -f step and use --format "{{.Names}}\t{{.Status}}".

License

MIT — see LICENSE for details.

Star History

<a href="https://www.star-history.com/?repos=internlm%2FWildClawBench&type=date&legend=top-left"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=internlm/WildClawBench&type=date&theme=dark&legend=top-left&sealedtoken=uOE3PXZranBUxC9q3pvz0oRRTksC3hKub6Y69tgdPgf3FZ0kurMi1a4jofUYZLkXYgMKgc2OvBvpJ3kCRHiVM1PzjpMKlxhLwnGwYKBzeJewVmL2KNolg" /> <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=internlm/WildClawBench&type=date&legend=top-left&sealedtoken=uOE3PXZranBUxC9q3pvz0oRRTksC3hKub6Y69tgdPgf3FZ0kurMi1a4jofUYZLkXYgMKgc2OvBvpJ3kCRHiVM1PzjpMKlxhLwnGwYKBzeJewVmL2KNolg" /> <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=internlm/WildClawBench&type=date&legend=top-left&sealedtoken=uOE3PXZranBUxC9q3pvz0oRRTksC3hKub6Y69tgdPgf3FZ0kurMi1a4jofUYZLkXYgMKgc2OvBvpJ3kCRHiVM1PzjpMKlxhLwnGwYKBzeJewVmL2KNolg" /> </picture> </a>