AIGrounding/Diagram-MMU

Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

ECCV 2026

🏠 Homepage (coming soon) · 💻 Code · 📄 Paper (coming soon)

Diagram-MMU is a benchmark for evaluating Multimodal Large Language Models (MLLMs) on understanding, parsing, and editing scientific diagrams. It contains 3,744 curated diagrams (each with compilable source code) and 18,305 human-validated evaluation instances across six domains (charts, planar_geometry, 3d_shapes, graph_structures, chemistry, circuit_diagrams), over three tasks: diagram-to-code parsing (D2C-P), diagram-to-code editing (D2C-E), and diagram question answering (DQA).

Each config provides two splits: `test` (full benchmark) and `testmini` (a class-balanced 50-per-domain subset, 300 diagrams, for quick development). All ground truth is public, so evaluation runs locally with no submission.

Data Distribution

<p align="center"> <img src="assets/data_statistics.png" width="900" alt="Per-domain statistics and task distribution"> </p>

Main Results

<p align="center"> <img src="assets/main_results.png" width="900" alt="Main results of 12 MLLMs on Diagram-MMU"> </p>

Usage

from datasets import load_dataset

d2cp = load_dataset("AIGrounding/Diagram-MMU", "d2c-p", split="test")       # parsing
dqa  = load_dataset("AIGrounding/Diagram-MMU", "dqa", split="testmini")     # QA, dev subset
diagrams = load_dataset("AIGrounding/Diagram-MMU", "diagrams", split="test")

ex = d2cp[0]
ex["image"]   # PIL.Image (decoded automatically)
ex["code"]    # ground-truth source

Evaluation

All ground truth is public, so evaluation runs locally — no submission. The official evaluation code (object / code / image metrics for D2C-P & D2C-E, and the rule-based + LLM-as-judge pipeline for DQA) will be released separately:

• —Evaluation repository: github.com/AIGrounding/Diagram-MMU

License & Citation

Released under the Apache License 2.0. Source code is collected from official package handbooks (PGFPlots, CircuiTikZ, TKZ-Euclide, ChemFig, TikZ-Network) and community resources (texample.net, TeX Stack Exchange, GitHub tikz_favorites); upstream sources may carry their own terms. All annotations are original to this work.

@article{bo2026diagrammmu,
  title   = {Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams},
  author  = {Bo, Weihao and Zhang, Shan and Sun, Yanpeng and Liu, Jie and Yao, Yongke and Du, Jinhao and He, Wei and Zou, Kai and Li, Zechao and Wang, Jingdong},
  journal = {arXiv preprint arXiv:TODO},
  year    = {2026}
}