chartanno/ChartAnno

1---2license: cc-by-nc-4.03language:4- en5pretty_name: ChartAnno6tags:7- benchmark8- chart annotation9- code generation10- multimodal11- matplotlib12- visualization13task_categories:14- text-generation15- image-to-text16size_categories:17- 1K<n<10K18arxiv: 2608.0346419configs:20- config_name: 1_python_code_image21  data_files:22  - split: train23    path: 1_python_code_image/train-*24- config_name: 2_python_code_only25  data_files:26  - split: train27    path: 2_python_code_only/train-*28- config_name: 3_python_image_only29  data_files:30  - split: train31    path: 3_python_image_only/train-*32- config_name: 4_d3_code_only33  data_files:34  - split: train35    path: 4_d3_code_only/train-*36- config_name: 5_svg_code_only37  data_files:38  - split: train39    path: 5_svg_code_only/train-*40dataset_info:41- config_name: 1_python_code_image42  features:43  - name: id44    dtype: string45  - name: category46    dtype: string47  - name: sample_id48    dtype: string49  - name: level50    dtype: string51  - name: input_type52    dtype: string53  - name: instruction54    dtype: string55  - name: GT w/o anno code56    dtype: string57  - name: gt_wo_anno_chart58    dtype: image59  - name: GT code60    dtype: string61  - name: gt_chart62    dtype: image63  splits:64  - name: train65    num_examples: 360066- config_name: 2_python_code_only67  features:68  - name: id69    dtype: string70  - name: category71    dtype: string72  - name: sample_id73    dtype: string74  - name: level75    dtype: string76  - name: input_type77    dtype: string78  - name: instruction79    dtype: string80  - name: GT w/o anno code81    dtype: string82  - name: gt_wo_anno_chart83    dtype: image84  - name: GT code85    dtype: string86  - name: gt_chart87    dtype: image88  splits:89  - name: train90    num_examples: 360091- config_name: 3_python_image_only92  features:93  - name: id94    dtype: string95  - name: category96    dtype: string97  - name: sample_id98    dtype: string99  - name: level100    dtype: string101  - name: input_type102    dtype: string103  - name: instruction104    dtype: string105  - name: GT w/o anno code106    dtype: string107  - name: gt_wo_anno_chart108    dtype: image109  - name: GT code110    dtype: string111  - name: gt_chart112    dtype: image113  splits:114  - name: train115    num_examples: 3600116- config_name: 4_d3_code_only117  features:118  - name: id119    dtype: string120  - name: category121    dtype: string122  - name: sample_id123    dtype: string124  - name: level125    dtype: string126  - name: input_type127    dtype: string128  - name: instruction129    dtype: string130  - name: GT w/o anno code131    dtype: string132  - name: gt_wo_anno_chart133    dtype: image134  - name: GT code135    dtype: string136  - name: gt_chart137    dtype: image138  splits:139  - name: train140    num_examples: 360141- config_name: 5_svg_code_only142  features:143  - name: id144    dtype: string145  - name: category146    dtype: string147  - name: sample_id148    dtype: string149  - name: level150    dtype: string151  - name: input_type152    dtype: string153  - name: instruction154    dtype: string155  - name: GT w/o anno code156    dtype: string157  - name: gt_wo_anno_chart158    dtype: image159  - name: GT code160    dtype: string161  - name: gt_chart162    dtype: image163  splits:164  - name: train165    num_examples: 360166---167 168<p align="center">169  <img src="assets/logo.png" width="130" alt="ChartAnno logo">170</p>171 172<h1 align="center">ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation</h1>173 174<p align="center">175  <a href="https://chartanno.github.io/"><img src="https://img.shields.io/badge/Homepage-chartanno.github.io-teal.svg" alt="Homepage"></a>176  &nbsp;177  <a href="https://arxiv.org/abs/2608.03464"><img src="https://img.shields.io/badge/arXiv-2608.03464-b31b1b.svg" alt="arXiv"></a>178  &nbsp;179  <a href="https://github.com/ChartAnno/ChartAnno"><img src="https://img.shields.io/badge/GitHub-ChartAnno-24292e.svg?logo=github" alt="GitHub"></a>180  &nbsp;181  <a href="https://huggingface.co/datasets/chartanno/ChartAnno"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-chartanno%2FChartAnno-ffd21f.svg" alt="HuggingFace Dataset"></a>182  &nbsp;183  <a href="https://creativecommons.org/licenses/by-nc/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC%204.0-blue.svg" alt="License"></a>184</p>185 186<h3 align="center">The official dataset repository of ChartAnno</h3>187 188<p align="center">189  <b>1,200</b> real-world charts &nbsp;·&nbsp; <b>3,600</b> instructions &nbsp;·&nbsp; <b>10,800</b> instances (+720 D3/SVG) &nbsp;·&nbsp; <b>3</b> representations &nbsp;·&nbsp; <b>17</b> chart types190</p>191 192## 1. Data Overview193 194Annotations are essential to communicative visualization, helping explain data, emphasize key findings, and guide attention. **ChartAnno** is a benchmark for evaluating multimodal large language models (MLLMs) on **chart annotation generation**: given an existing chart (as executable code, optionally with the rendered image) and an annotation instruction, the model generates executable code that adds the requested annotations.195 196ChartAnno includes **1,200 real-world charts**, each with a paired **annotated / unannotated ground-truth** (code + rendered image) pair. Every chart is paired with annotation instructions across three levels of specificity — **Intent**, **Operation**, and **Implementation** — yielding **3,600 instruction instances**. The primary benchmark tests **two input settings** (`Input: code` and `Input: code+Image`) for **7,200 model input instances**; a supplementary `Input: Image` setting adds **3,600 image-only instances**, and a 120-chart **D3.js / SVG extension** adds **720** more. Across the benchmark the same task is instantiated in **three representations** — Python, D3.js, and SVG.197 198<p align="center">199  <img src="assets/main-chart.png" width="95%" alt="ChartAnno task design">200</p>201<p align="center"><sub>Task design of ChartAnno. Primary chart inputs are Code or Code + Chart Image, with annotation instructions ranging from abstract Intent through Operation to concrete Implementation. For each chart input–instruction combination, the MLLM generates annotated code and its rendered chart, evaluated with reference to the unannotated code, instruction, and annotated ground truth.</sub></p>202 203## 2. Task Formats204 205The dataset is organized into **configs**, one per input setting:206 207<table align="center">208<tr>209  <th align="center">Config</th>210  <th align="center">Input setting</th>211  <th align="center">Model input</th>212  <th align="center">Ground truth</th>213</tr>214<tr>215  <td align="center"><code>1_python_code_image</code></td>216  <td align="center">Input: code+Image</td>217  <td align="center">Instruction + unannotated chart code + chart image</td>218  <td align="center">Annotated code + annotated chart</td>219</tr>220<tr>221  <td align="center"><code>2_python_code_only</code></td>222  <td align="center">Input: code</td>223  <td align="center">Instruction + unannotated chart code</td>224  <td align="center">Annotated code + annotated chart</td>225</tr>226<tr>227  <td align="center"><code>3_python_image_only</code></td>228  <td align="center">Input: Image</td>229  <td align="center">Instruction + unannotated chart image</td>230  <td align="center">Annotated code + annotated chart</td>231</tr>232<tr>233  <td align="center"><code>4_d3_code_only</code></td>234  <td align="center">Input: code (D3.js)</td>235  <td align="center">Instruction + unannotated D3.js code</td>236  <td align="center">Annotated D3.js code + annotated chart</td>237</tr>238<tr>239  <td align="center"><code>5_svg_code_only</code></td>240  <td align="center">Input: code (SVG)</td>241  <td align="center">Instruction + unannotated SVG document</td>242  <td align="center">Annotated SVG document + annotated chart</td>243</tr>244</table>245 246Each instance carries one of three instruction levels:247 248| Level | Description |249| --- | --- |250| `intent` | Abstract communicative goal, e.g. "emphasize the strongest market". The model must infer what to annotate. |251| `operation` | Concrete annotation operations, e.g. "add an arrow pointing to the peak and a text label with its value". |252| `implementation` | Fully specified implementation details, e.g. exact label text, colors, and positions. |253 254An example pair from the `1_python_code_image` config (`Area_33`): the model receives the unannotated chart on the left (with its code) and produces the annotated version on the right.255 256<table align="center">257<tr>258  <th align="center">Input: GT w/o anno chart</th>259  <th align="center">Expected output: GT chart</th>260</tr>261<tr>262  <td align="center"><img src="assets/example_unannotated.jpg" width="400" alt="Unannotated input chart"></td>263  <td align="center"><img src="assets/example_annotated.jpg" width="400" alt="Annotated ground-truth chart"></td>264</tr>265</table>266 267### D3.js / SVG extension (code-only)268 269Besides the main Python benchmark, the dataset ships a **D3.js and SVG extension**: **120 real-world charts** (14 chart types), each paired with instructions across the same three levels — **360 rows per backend** (`4_d3_code_only`, `5_svg_code_only`). Rows follow the same schema as `2_python_code_only`; ids carry a `_d3` / `_svg` suffix, and the code files (`.js` / `.svg`) plus full-resolution images are included in `chartanno_d3_svg.tar.gz`.270 271## 3. Data Statistics272 273The statistics below are computed on the main Python benchmark; the D3/SVG extension follows the same construction.274 275**Overall scale**276 277| Statistic | Value |278| --- | --- |279| GT pair and GT w/o anno pair | 1,200 |280| Instruction instances | 3,600 |281| Annotation elements | 25,772 |282 283**Annotation complexity**284 285| Statistic | Value |286| --- | --- |287| Avg. elements per chart | 21.48 |288| Std. elements per chart | 35.32 |289| Avg. annotation types per chart | 2.17 |290| Std. annotation types per chart | 1.04 |291 292**Instruction length (words)**293 294| Level | Avg. | Std. |295| --- | --- | --- |296| Intent | 66.39 | 52.97 |297| Operation | 95.06 | 63.57 |298| Implementation | 108.74 | 79.78 |299 300**Code length (tokens)**301 302| Code | Avg. | Std. |303| --- | --- | --- |304| Annotated | 1535.68 | 972.38 |305| Unannotated | 1075.22 | 790.45 |306| Increment | 460.46 | 385.17 |307 308**Code token increment across representations (120-chart extension)**309 310On the same 120 charts, the annotation increment differs substantially across representations:311 312| Representation | Avg. code token increment | Relative to Python |313| --- | --- | --- |314| Python | 395.0 | 1.00× |315| D3.js | 1374.2 | 3.48× |316| SVG | 2612.7 | 6.61× (1.90× D3) |317 318**Chart type distribution (17 types)**319 320| Type | Multi | Line | Bar | Scatter | Errorpoint | Heatmap | Pie | Area | Histogram |321| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |322| Count | 303 | 247 | 213 | 94 | 47 | 44 | 44 | 33 | 32 |323 324| Type | Combination | Violin | Radar | Density | Treemap | Box | Contour | Dot | Total |325| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |326| Count | 31 | 20 | 19 | 19 | 17 | 16 | 15 | 6 | 1,200 |327 328## 4. Download Link329 330This dataset is gated: click **Request access** on the repo page first. Once granted, you can download the whole dataset with a read token ([settings/tokens](https://huggingface.co/settings/tokens)):331 332```shell333pip install -U huggingface_hub334hf login --token $HF_TOKEN        # or: hf auth login335 336mkdir ChartAnno && cd ChartAnno   # cd to the target directory337hf download chartanno/ChartAnno chartanno.tar.gz --repo-type dataset --local-dir .338tar -xzvf chartanno.tar.gz339```340 341The optional D3/SVG extension ships as a separate archive:342 343```shell344hf download chartanno/ChartAnno chartanno_d3_svg.tar.gz --repo-type dataset --local-dir .345tar -xzvf chartanno_d3_svg.tar.gz   # extracts to ./d3_svg_data346```347 348The file structure of the dataset is as follows:349 350```text351data/352├── input_code.jsonl              # Input: code task, 3,600 rows353├── input_code_image.jsonl        # Input: code+Image task, 3,600 rows354├── input_image_only.jsonl        # Input: Image task, 3,600 rows355├── manifest.json                 # row-count summary and schema description356├── README.md357└── images/358    ├── GT_chart/                 # 1,200 annotated ground-truth charts (jpg)359    │   ├── Area/  Bar/  ...      # one subdirectory per chart type (17 types)360    └── GT_w_o_anno_chart/        # 1,200 unannotated charts, same layout361```362 363The [Dataset Viewer](https://huggingface.co/datasets/chartanno/ChartAnno/viewer/1_python_code_image/train) above serves the **full dataset**: all 3,600 rows per config are loadable via `load_dataset("chartanno/ChartAnno", "<config>")` (configs `1_python_code_image`, `2_python_code_only`, `3_python_image_only` are the main benchmark; `4_d3_code_only` / `5_svg_code_only` are the extension). Chart images in the viewer parquet are embedded as compressed JPEG thumbnails for fast previewing; the full-resolution images are included in `chartanno.tar.gz`.364 365## 5. Data Fields366 367We take an instance from the `1_python_code_image` config as an example (`Area_1_intent_code_image`, long values truncated):368 369```json370{371  "id": "Area_1_intent_code_image",372  "category": "Area",373  "sample_id": "Area_1",374  "level": "intent",375  "input_type": "Input: code+Image",376  "instruction": "You are an expert in chart annotation and Python visualization. I have created a figure but have not added any annotations yet. ...",377  "GT w/o anno code": "import matplotlib.pyplot as plt\nimport numpy as np\nmonths = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']\nvisitors = [12, 120, 229, 990, 4104, 3250, 5720, 43152, ...",378  "GT w/o anno chart": "data/images/GT_w_o_anno_chart/Area/Area_1.jpg",379  "GT code": "import matplotlib.pyplot as plt\nimport numpy as np\nmonths = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']\nvisitors = [12, 120, 229, 990, 4104, 3250, 5720, 43152, ...",380  "GT chart": "data/images/GT_chart/Area/Area_1.jpg"381}382```383 384Details of the data fields are as follows:385 386| Field | Description |387| --- | --- |388| `id` | Stable row id: `<sample_id>_<level>_code` or `<sample_id>_<level>_code_image`. |389| `category` | Chart type, one of 17 types (e.g. `Area`, `Bar`, `Line`). |390| `sample_id` | Source chart id, `<Category>_<n>` (e.g. `Area_1`). |391| `level` | Instruction level: `intent`, `operation`, or `implementation`. |392| `input_type` | `Input: code` or `Input: code+Image`. |393| `instruction` | The full model input prompt: task instruction with the unannotated code embedded. |394| `GT w/o anno code` | Ground-truth chart code before annotation. |395| `GT w/o anno chart` | Path to the unannotated chart image; always `null` in the `2_python_code_only` config. |396| `GT code` | Ground-truth annotated chart code. |397| `GT chart` | Path to the annotated ground-truth chart image. |398 399Image paths are relative to the repository root of the extracted `data/` folder. In the viewer parquet the two image fields are named `gt_chart` / `gt_wo_anno_chart` (space-free, for Dataset Viewer compatibility) and are embedded as compressed thumbnails of these files.400 401## 6. Evaluation402 403ChartAnno comes with a self-contained evaluation pipeline (see the [GitHub repository](https://github.com/ChartAnno/ChartAnno)): it renders the generated code into charts and scores them with rule-based metrics (execution success, chart fidelity, annotation matching, color matching) and LLM-judged metrics (semantic faithfulness, semantic clarity, visual clarity, annotation organization quality, attention guidance). The final aggregate scores are:404 405| Aggregate | Inputs |406| --- | --- |407| Structural compliance | chart fidelity (+ annotation & color matching for operation/implementation levels) |408| Semantic consistency | semantic faithfulness, semantic clarity |409| Design effectiveness | visual clarity, annotation organization quality, attention guidance |410 411## 7. Citation412 413If you find ChartAnno useful, please consider citing our paper:414 415```bibtex416@article{chen2026chartanno,417      title={ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation},418      author={Zhenghan Chen and Zekai Shao and Lidan Tan and Xin Lin and Xingchen Zeng and Yi Shan and Ziyue Lin and Xiaoliang Fu and Xinyuan Liu and Yuetong Guo and Fen Wang and Bongshin Lee and Siming Chen},419      year={2026},420      journal={arXiv preprint arXiv:2608.03464},421}422```423