DangIT02/image-to-mermaid-v3

Image-to-Mermaid Dataset v3.0

Task: Given a rendered diagram image, generate the equivalent Mermaid source code that reproduces it.

This is a synthetic, multi-type extension of an earlier flowchart-only release (2,569 samples, tied to an accepted MAPR2026 IEEE paper on flowchart QA). v3.0 covers 26 Mermaid diagram types across 3 difficulty tiers each.

Statistics

Difficulty tiers (defined by structural element count — node/edge counts and a composite difficulty score — not by text length):

By diagram type (26 types, sorted by total samples):

Structural ranges across the corpus: 0–72 nodes, 0–101 edges, composite difficulty score 0.09–8.0 (see stats.json for the exact per-split distributions).

Dataset Structure

v3.0/
├── train.json           # Training split (12,046 samples)
├── val.json              # Validation split (1,475 samples)
├── test.json             # Test split (1,578 samples)
├── images/               # 15,099 PNGs, one subfolder per diagram type
│   ├── flowchart/
│   │   ├── easy/
│   │   ├── medium/
│   │   └── hard/
│   ├── classDiagram/
│   │   ├── easy/ medium/ hard/
│   ├── ...               # 26 diagram-type folders total
├── stats.json             # Per-split / per-type / per-tier counts and structural stats
├── DATASET_INFO.md        # Generation-run metadata
├── README.md               # This file
└── LICENSE

Format (Qwen-VL conversation JSON)

Each record is a single-turn image-grounded conversation. Real example pulled from train.json:

{
  "id": "classDiagram_0933",
  "conversations": [
    {
      "from": "user",
      "value": "Picture 1: <img>images/classDiagram/medium/classDiagram_0933_d96139.png</img>\nConvert this diagram to Mermaid code. Provide only the Mermaid syntax."
    },
    {
      "from": "assistant",
      "value": "classDiagram\n    class RideParticipant {\n        <<interface>>\n        +String participantId\n        +updateProfile() void\n    }\n    ...\n    Driver ..|> RideParticipant\n    ..."
    }
  ]
}

The image path is always relative to the dataset root (images/{type}/{tier}/{file}.png). The instruction sentence is drawn from a small pool of paraphrase templates (e.g. "Convert this diagram to Mermaid code.", "Provide the Mermaid code for this diagram.", "Analyze this diagram and output the equivalent Mermaid code.") to reduce instruction overfitting; all templates are English.

Images are PNGs rendered directly by mermaid-cli and are not resized to a fixed resolution — dimensions vary per diagram (driven by node/edge count and layout), with black diagram content on a white background.

Generation & Validation Methodology

Samples were generated via Batch API across three LLM providers — Anthropic Claude, OpenAI GPT, and Google Gemini — routed to different diagram-type/difficulty cells so no single provider dominates any one cell. Every candidate was rendered with mermaid-cli (mmdc) and had to pass a render-success gate plus a "ghost render" check (mmdc can exit 0 while silently emitting a fixed-size placeholder image instead of the real diagram). Surviving samples then went through four-layer deduplication: exact match (SHA256 on normalized Mermaid code), near-duplicate (MinHash-LSH, Jaccard ≥ 0.7), semantic (SemDeDup, cosine similarity ≥ 0.9), and structural (Weisfeiler-Lehman graph-hash isomorphism for graph-shaped diagram types).

Renderer pin (reproducibility contract): all images in this dataset were rendered with mermaid-cli (mmdc) v11.15.0. This version is part of the dataset's definition — re-rendering the same Mermaid code with a different mmdc version can produce visually different output, so any downstream evaluation that compares against these images should pin the same renderer version.

Known data quality note

An image-integrity spot check performed while preparing this card found 718/15,099 images (4.76%) were a byte-identical 512×109px placeholder ("ghost render") instead of the real diagram, concentrated in treemap (300/300) and architecture (347/493), with smaller leakage into gitGraph, treeview, and radar. Root cause was a handful of distinct Mermaid syntax patterns this project's pinned mmdc 11.15.0 silently fails to render (e.g. id["Label": value] leaf syntax in treemap-beta, block-style group ... end declarations in architecture-beta) plus a few genuine content bugs in the generated code (a tag: statement not attached to a commit, a spelled-out number where radar-beta expects a digit, a leaf label colliding with the title keyword in treeview-beta). All but 15 samples (0.10% of the dataset) were repaired by rewriting the affected Mermaid source to an equivalent, renderer-compatible form and re-rendering with the same pinned mmdc version — both the code (assistant turn) and image were updated together so they stay consistent. The 15 remaining known-broken images are: treemap (6), gitGraph (6), architecture (1), radar (1), treeview (1); each needs individual inspection rather than a general fix.

Usage

from datasets import load_dataset

ds = load_dataset("<namespace>/<repo_name>")
sample = ds["train"][0]
# sample["conversations"][0]["value"] -> instruction with image tag
# sample["conversations"][1]["value"] -> ground-truth Mermaid code

Note: this repo stores raw JSON + an images/ folder rather than a datasets-native schema (e.g. Parquet with embedded image bytes), matching the Qwen-VL fine-tuning format it was built for. load_dataset with the default loader may not parse the conversation/image-path structure automatically — treat train.json/val.json/test.json plus images/ as the source of truth and load them directly with json.load(...) if you need the raw records.

Intended Uses

• —Vision-language model fine-tuning for diagram understanding (image → structured code)
• —Diagram-to-code / OCR-style benchmarks across a broad set of Mermaid diagram types
• —Multimodal structural-reasoning evaluation

License

MIT — see LICENSE. This dataset extends a previously released flowchart-only dataset (also MIT-licensed) associated with an accepted MAPR2026 IEEE paper.