haizelabs/calligraphy-bench

Calligraphy Bench

Can AI agents draw Chinese calligraphy stroke-by-stroke?

88 of the hardest Chinese characters from 20 master calligraphers, drawn by 4 frontier AI agents on a tldraw canvas using programmatic stroke commands. Each output is evaluated on stroke correctness, stroke ordering, visual fidelity, and pairwise preference ranking.

Task

Given a reference calligraphy image (provided as a vision message — no file access), an AI agent must:

1. 1.Draw the character stroke-by-stroke via MCP tools (create_stroke, update_stroke, etc.)
2. 2.Control position, pressure (thick-thin variation), and size for each stroke
3. 3.Capture and submit the final output via save_document

No tracing, no bitmap copying, no image generation models — only programmatic strokes on a tldraw canvas through MCP tools.

Models Evaluated

Evaluation Metrics

• —Stroke Recall: What fraction of reference strokes does the agent reproduce? (vision judge)
• —Stroke Order: Of matched strokes, what fraction are drawn in correct canonical order? (vision judge)
• —Fidelity Verdict: 1-5 scale comparing visual similarity to reference (vision judge)
• —Pairwise Win Rate: Head-to-head comparison using MJ1 judge (Tinker, round-robin all pairs)

Data Fields

• —task_id: Unique identifier for the character+calligrapher combination
• —character: The Chinese character
• —pinyin: Romanized pronunciation
• —definition: English meaning
• —stroke_count: Number of canonical strokes
• —style_difficulty: 1-5 difficulty rating
• —calligrapher: Name of the master calligrapher
• —reference_image: Original calligraphy by the master calligrapher

Each row is one character (88 total). This dataset contains only task inputs — model outputs and evaluation results are available on the benchmark website.

Usage

from datasets import load_dataset

ds = load_dataset("haizelabs/calligraphy-bench", split="test")
print(f"{len(ds)} characters")

# Browse tasks
for row in ds:
    print(f"{row['character']} ({row['pinyin']}) — {row['calligrapher']}, {row['stroke_count']} strokes")

Harness

Agents are run via OpenHands SDK with MCP tools on a tldraw canvas. To submit your model for evaluation, contact Haize Labs.

Citation

@misc{calligraphy-bench-2026,
  title={Calligraphy Bench: Can AI Write Chinese Calligraphy?},
  year={2026},
  url={https://huggingface.co/datasets/haizelabs/calligraphy-bench}
}