Datasetpublic

chartanno/ChartAnno

ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation         The official dataset repository of ChartAnno 1,200 real-world charts  ·  3,600 instructions  ·  10,800 instances (+720 D3/SVG)  ·  3 representations  ·  17 chart types 1. Data Overview Annotations are essential to communicative visualization, helping explain data, emphasize key findings… See the full description on the dataset page: https://huggingface.co/datasets/chartanno/ChartAnno.

sourceHugging Facecc-by-nc-4.0updated 5d agoView on Hugging Face

1likes609downloads

Dataset Card

<h1 align="center">ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation</h1>

<h3 align="center">The official dataset repository of ChartAnno</h3>

<p align="center"> <b>1,200</b> real-world charts  ·  <b>3,600</b> instructions  ·  <b>10,800</b> instances (+720 D3/SVG)  ·  <b>3</b> representations  ·  <b>17</b> chart types </p>

1. Data Overview

Annotations are essential to communicative visualization, helping explain data, emphasize key findings, and guide attention. ChartAnno is a benchmark for evaluating multimodal large language models (MLLMs) on chart annotation generation: given an existing chart (as executable code, optionally with the rendered image) and an annotation instruction, the model generates executable code that adds the requested annotations.

ChartAnno includes 1,200 real-world charts, each with a paired annotated / unannotated ground-truth (code + rendered image) pair. Every chart is paired with annotation instructions across three levels of specificity — Intent, Operation, and Implementation — yielding 3,600 instruction instances. The primary benchmark tests two input settings (Input: code and Input: code+Image) for 7,200 model input instances; a supplementary Input: Image setting adds 3,600 image-only instances, and a 120-chart D3.js / SVG extension adds 720 more. Across the benchmark the same task is instantiated in three representations — Python, D3.js, and SVG.

<p align="center"> <img src="assets/main-chart.png" width="95%" alt="ChartAnno task design"> </p> <p align="center"><sub>Task design of ChartAnno. Primary chart inputs are Code or Code + Chart Image, with annotation instructions ranging from abstract Intent through Operation to concrete Implementation. For each chart input–instruction combination, the MLLM generates annotated code and its rendered chart, evaluated with reference to the unannotated code, instruction, and annotated ground truth.</sub></p>

2. Task Formats

The dataset is organized into configs, one per input setting:

<table align="center"> <tr> <th align="center">Config</th> <th align="center">Input setting</th> <th align="center">Model input</th> <th align="center">Ground truth</th> </tr> <tr> <td align="center"><code>1pythoncodeimage</code></td> <td align="center">Input: code+Image</td> <td align="center">Instruction + unannotated chart code + chart image</td> <td align="center">Annotated code + annotated chart</td> </tr> <tr> <td align="center"><code>2pythoncodeonly</code></td> <td align="center">Input: code</td> <td align="center">Instruction + unannotated chart code</td> <td align="center">Annotated code + annotated chart</td> </tr> <tr> <td align="center"><code>3pythonimageonly</code></td> <td align="center">Input: Image</td> <td align="center">Instruction + unannotated chart image</td> <td align="center">Annotated code + annotated chart</td> </tr> <tr> <td align="center"><code>4d3codeonly</code></td> <td align="center">Input: code (D3.js)</td> <td align="center">Instruction + unannotated D3.js code</td> <td align="center">Annotated D3.js code + annotated chart</td> </tr> <tr> <td align="center"><code>5svgcode_only</code></td> <td align="center">Input: code (SVG)</td> <td align="center">Instruction + unannotated SVG document</td> <td align="center">Annotated SVG document + annotated chart</td> </tr> </table>

Each instance carries one of three instruction levels:

Level	Description
`intent`	Abstract communicative goal, e.g. "emphasize the strongest market". The model must infer what to annotate.
`operation`	Concrete annotation operations, e.g. "add an arrow pointing to the peak and a text label with its value".
`implementation`	Fully specified implementation details, e.g. exact label text, colors, and positions.

An example pair from the 1_python_code_image config (Area_33): the model receives the unannotated chart on the left (with its code) and produces the annotated version on the right.

<table align="center"> <tr> <th align="center">Input: GT w/o anno chart</th> <th align="center">Expected output: GT chart</th> </tr> <tr> <td align="center"><img src="assets/exampleunannotated.jpg" width="400" alt="Unannotated input chart"></td> <td align="center"><img src="assets/exampleannotated.jpg" width="400" alt="Annotated ground-truth chart"></td> </tr> </table>

D3.js / SVG extension (code-only)

Besides the main Python benchmark, the dataset ships a D3.js and SVG extension: 120 real-world charts (14 chart types), each paired with instructions across the same three levels — 360 rows per backend (4_d3_code_only, 5_svg_code_only). Rows follow the same schema as 2_python_code_only; ids carry a _d3 / _svg suffix, and the code files (.js / .svg) plus full-resolution images are included in chartanno_d3_svg.tar.gz.

3. Data Statistics

The statistics below are computed on the main Python benchmark; the D3/SVG extension follows the same construction.

Overall scale

Statistic	Value
GT pair and GT w/o anno pair	1,200
Instruction instances	3,600
Annotation elements	25,772

Annotation complexity

Statistic	Value
Avg. elements per chart	21.48
Std. elements per chart	35.32
Avg. annotation types per chart	2.17
Std. annotation types per chart	1.04

Instruction length (words)

Level	Avg.	Std.
Intent	66.39	52.97
Operation	95.06	63.57
Implementation	108.74	79.78

Code length (tokens)

Code	Avg.	Std.
Annotated	1535.68	972.38
Unannotated	1075.22	790.45
Increment	460.46	385.17

Code token increment across representations (120-chart extension)

On the same 120 charts, the annotation increment differs substantially across representations:

Representation	Avg. code token increment	Relative to Python
Python	395.0	1.00×
D3.js	1374.2	3.48×
SVG	2612.7	6.61× (1.90× D3)

Chart type distribution (17 types)

Type	Multi	Line	Bar	Scatter	Errorpoint	Heatmap	Pie	Area	Histogram
Count	303	247	213	94	47	44	44	33	32

Type	Combination	Violin	Radar	Density	Treemap	Box	Contour	Dot	Total
Count	31	20	19	19	17	16	15	6	1,200

4. Download Link

This dataset is gated: click Request access on the repo page first. Once granted, you can download the whole dataset with a read token (settings/tokens):

shell

pip install -U huggingface_hub
hf login --token $HF_TOKEN        # or: hf auth login

mkdir ChartAnno && cd ChartAnno   # cd to the target directory
hf download chartanno/ChartAnno chartanno.tar.gz --repo-type dataset --local-dir .
tar -xzvf chartanno.tar.gz

The optional D3/SVG extension ships as a separate archive:

shell

hf download chartanno/ChartAnno chartanno_d3_svg.tar.gz --repo-type dataset --local-dir .
tar -xzvf chartanno_d3_svg.tar.gz   # extracts to ./d3_svg_data

The file structure of the dataset is as follows:

text

data/
├── input_code.jsonl              # Input: code task, 3,600 rows
├── input_code_image.jsonl        # Input: code+Image task, 3,600 rows
├── input_image_only.jsonl        # Input: Image task, 3,600 rows
├── manifest.json                 # row-count summary and schema description
├── README.md
└── images/
    ├── GT_chart/                 # 1,200 annotated ground-truth charts (jpg)
    │   ├── Area/  Bar/  ...      # one subdirectory per chart type (17 types)
    └── GT_w_o_anno_chart/        # 1,200 unannotated charts, same layout

The Dataset Viewer above serves the full dataset: all 3,600 rows per config are loadable via load_dataset("chartanno/ChartAnno", "<config>") (configs 1_python_code_image, 2_python_code_only, 3_python_image_only are the main benchmark; 4_d3_code_only / 5_svg_code_only are the extension). Chart images in the viewer parquet are embedded as compressed JPEG thumbnails for fast previewing; the full-resolution images are included in chartanno.tar.gz.

5. Data Fields

We take an instance from the 1_python_code_image config as an example (Area_1_intent_code_image, long values truncated):

json

{
  "id": "Area_1_intent_code_image",
  "category": "Area",
  "sample_id": "Area_1",
  "level": "intent",
  "input_type": "Input: code+Image",
  "instruction": "You are an expert in chart annotation and Python visualization. I have created a figure but have not added any annotations yet. ...",
  "GT w/o anno code": "import matplotlib.pyplot as plt\nimport numpy as np\nmonths = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']\nvisitors = [12, 120, 229, 990, 4104, 3250, 5720, 43152, ...",
  "GT w/o anno chart": "data/images/GT_w_o_anno_chart/Area/Area_1.jpg",
  "GT code": "import matplotlib.pyplot as plt\nimport numpy as np\nmonths = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']\nvisitors = [12, 120, 229, 990, 4104, 3250, 5720, 43152, ...",
  "GT chart": "data/images/GT_chart/Area/Area_1.jpg"
}

Details of the data fields are as follows:

Field	Description
`id`	Stable row id: `<sample_id>_<level>_code` or `<sample_id>_<level>_code_image`.
`category`	Chart type, one of 17 types (e.g. `Area`, `Bar`, `Line`).
`sample_id`	Source chart id, `<Category>_<n>` (e.g. `Area_1`).
`level`	Instruction level: `intent`, `operation`, or `implementation`.
`input_type`	`Input: code` or `Input: code+Image`.
`instruction`	The full model input prompt: task instruction with the unannotated code embedded.
`GT w/o anno code`	Ground-truth chart code before annotation.
`GT w/o anno chart`	Path to the unannotated chart image; always `null` in the `2_python_code_only` config.
`GT code`	Ground-truth annotated chart code.
`GT chart`	Path to the annotated ground-truth chart image.

Image paths are relative to the repository root of the extracted data/ folder. In the viewer parquet the two image fields are named gt_chart / gt_wo_anno_chart (space-free, for Dataset Viewer compatibility) and are embedded as compressed thumbnails of these files.

6. Evaluation

ChartAnno comes with a self-contained evaluation pipeline (see the GitHub repository): it renders the generated code into charts and scores them with rule-based metrics (execution success, chart fidelity, annotation matching, color matching) and LLM-judged metrics (semantic faithfulness, semantic clarity, visual clarity, annotation organization quality, attention guidance). The final aggregate scores are:

Aggregate	Inputs
Structural compliance	chart fidelity (+ annotation & color matching for operation/implementation levels)
Semantic consistency	semantic faithfulness, semantic clarity
Design effectiveness	visual clarity, annotation organization quality, attention guidance

7. Citation

If you find ChartAnno useful, please consider citing our paper:

bibtex

@article{chen2026chartanno,
      title={ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation},
      author={Zhenghan Chen and Zekai Shao and Lidan Tan and Xin Lin and Xingchen Zeng and Yi Shan and Ziyue Lin and Xiaoliang Fu and Xinyuan Liu and Yuetong Guo and Fen Wang and Bongshin Lee and Siming Chen},
      year={2026},
      journal={arXiv preprint arXiv:2608.03464},
}