Sanjue-Logic/ai-design-benchmark

AI Design Benchmark

English | 中文

English

A reproducible benchmark framework for evaluating AI design tools across 7 design scenarios

   

News

• —2026-04: v1.1 released — added reference images (53 files), bilingual tasks, CI pipeline, and 9 scoring tests; paper PDF published as Release asset
• —2026-04: Round 1 results published — Lovart 56% (95% CI: 47%–64%), Jimeng 27%, Roboneo 19% across 7 scenes

Overview

This project provides a reproducible framework for cross-product evaluation of AI design tools, including:

• —7 design scenarios, 140 tasks in dataset — 115 tasks evaluated in Round 1 (Logo Design / IP Character Design / Badge & Trophy / Product Poster / Product Image Enhancement / Online Marketing / Offline Promotion)
• —Double-blind SbS (Side-by-Side) voting: evaluators pick the best output without knowing which product generated it
• —4-layer aggregation algorithm: Vote → Question → Task → Scene, ensuring full reproducibility
• —Two-step confidence quantification: Ticket concentration (≥20% / 10–20% / <10%) + Bootstrap 95% CI
• —Round 1 empirical results: cross-product comparison data for 3 publicly available AI design tools (Lovart / Jimeng / Roboneo)

Full paper → `paper/` | Methodology → `methodology/` | Task dataset → `dataset/`

Quick Start

Requirements

Python >= 3.9
pip install -r tools/requirements.txt

Reproduce Round 1 results

git clone https://github.com/SanJueLogic/ai-design-benchmark
cd ai-design-benchmark
python tools/scoring.py --input results/round-1/raw-votes.csv --output results/round-1/aggregated-repro.csv
python tools/confidence.py --input results/round-1/raw-votes.csv

Run your own evaluation round → `docs/how-to-run-a-round.md`

Repository Structure

ai-design-benchmark/
├── paper/                        # Full paper (Chinese Markdown + PDF)
├── methodology/                  # Methodology docs (metrics / scoring / evaluator protocol)
├── dataset/
│   └── v1.0/
│       ├── tasks.json            # Structured task dataset (140 tasks; 115 evaluated in Round 1)
│       ├── scenes/               # Per-scene task books (Markdown)
│       └── reference-images/     # Reference images for edit-type tasks
├── results/
│   └── round-1/
│       ├── raw-votes.csv         # Raw votes (evaluators anonymized)
│       ├── aggregated.csv        # Aggregated win-rate results
│       └── analysis-reports/     # Scene-level analysis reports
├── tools/
│   ├── scoring.py                # 4-layer aggregation scoring script
│   ├── confidence.py             # Confidence & Bootstrap CI calculator
│   └── sbs-template/             # SbS voting web template
└── docs/
    ├── how-to-run-a-round.md     # How to run an evaluation round
    ├── how-to-add-a-product.md   # How to add a new product
    └── how-to-contribute.md      # How to contribute new tasks

Round 1 Results Summary

Data collected: March–April 2026 | Method: double-blind SbS ranking | Evaluators: 10 (5 design / 4 engineering / 1 ML)

Full results and confidence explanations → `results/round-1/`

Citation

If this framework is helpful to your research, please cite:

@techreport{liu2026aidesignbenchmark,
  title     = {AI Design Tool Evaluation Benchmark v1.0: A Systematic Cross-Product Evaluation Framework for Enterprise Design Scenarios},
  author    = {Liu, Sanjue},
  year      = {2026},
  institution = {Independent Research},
  url       = {https://github.com/SanJueLogic/ai-design-benchmark}
}

Contributing

Contributions of new tasks or scenes via Pull Request are welcome. See `docs/how-to-contribute.md`

Contact

For questions, collaboration, or citation inquiries:

📧 842559943@qq.com

License

• —Data & documentation (paper, task dataset, results): CC BY 4.0 — free to use with attribution
• —Code (scripts under tools/): MIT

中文

一套面向 AI 设计工具的系统性评测框架

简介

本项目提供一套可复现的 AI 设计工具横向评测方法论，包含：

• —7 大设计场景、140 条任务（数据集总量），Round 1 实评 115 条（Logo 设计 / IP 形象设计 / 徽章奖杯设计 / 商品海报设计 / 商品图片美化 / 线上营销活动 / 线下推广物料）
• —双盲 SbS（Side-by-Side）投票机制：评测员在不知道图片来源的情况下逐题选出最优结果
• —四层聚合算法：投票 → 题目 → 任务 → 场景，保证结论可复现
• —两步置信度量化：票数集中度（≥20% / 10–20% / <10%）+ Bootstrap 95% CI
• —Round 1 实证结果：3 款公开 AI 设计工具（Lovart / 即梦 / Roboneo）横向对比数据

论文全文 → `paper/` | 方法论文档 → `methodology/` | 任务集 → `dataset/`

快速开始

环境要求

Python >= 3.9
pip install -r tools/requirements.txt

复现 Round 1 结果

git clone https://github.com/SanJueLogic/ai-design-benchmark
cd ai-design-benchmark
python tools/scoring.py --input results/round-1/raw-votes.csv --output results/round-1/aggregated-repro.csv
python tools/confidence.py --input results/round-1/raw-votes.csv

运行你自己的评测 → `docs/how-to-run-a-round.md`

Round 1 结果摘要

数据采集时间：2026 年 3-4 月 | 评测方式：双盲 SbS | 评测员：10 人（设计 5 / 研发 4 / 算法 1）

引用

@techreport{liu2026aidesignbenchmark,
  title     = {AI 设计工具评测 Benchmark v1.0：面向企业级设计场景的系统性横向评测框架},
  author    = {刘三觉},
  year      = {2026},
  institution = {独立评测研究},
  url       = {https://github.com/SanJueLogic/ai-design-benchmark}
}

贡献

欢迎通过 Pull Request 提交新任务或新场景，详见 `docs/how-to-contribute.md`

联系

📧 842559943@qq.com

License

• —数据与文档（论文、任务集、评测结果）：CC BY 4.0 — 可自由使用，需注明来源
• —代码（tools/ 下的脚本）：MIT