DangIT02/qwen3vl-flowchart-to-mermaid-v5

Qwen3-VL-8B Flowchart → Mermaid (v5)

Fine-tuned Qwen3-VL-8B-Instruct for converting flowchart images into Mermaid code.

TL;DR

Quick start (vLLM)

vllm serve DangIT02/qwen3vl-flowchart-to-mermaid-v5 \
    --port 8000 \
    --max-model-len 8192 \
    --limit-mm-per-prompt image=1 \
    --gpu-memory-utilization 0.85

from openai import OpenAI
import base64

client = OpenAI(base_url="http://localhost:8000/v1", api_key="empty")

with open("flowchart.png", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="DangIT02/qwen3vl-flowchart-to-mermaid-v5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}},
            {"type": "text", "text": "Convert this flowchart diagram to Mermaid code."},
        ],
    }],
    temperature=0.0,
    max_tokens=2048,
)
print(response.choices[0].message.content)

Quick start (transformers)

from transformers import AutoModelForImageTextToText, AutoProcessor
import torch
from PIL import Image

model_id = "DangIT02/qwen3vl-flowchart-to-mermaid-v5"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("flowchart.png").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Convert this flowchart diagram to Mermaid code."},
    ],
}]
inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True).to("cuda")
output = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
print(processor.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Model details

• —Base model: unsloth/Qwen3-VL-8B-Instruct
• —Task: Image-to-code (flowchart image → Mermaid syntax)
• —Output direction: TD (top-down only)
• —Output syntax: 100% pipe syntax A -->|label| B (no alternate A -- label --> B)
• —Node ID style: Descriptive (Start, CheckInput) — not canonicalized

Training

Dataset

DangIT02/flowchart-to-mermaid_v2_2 (private, 2,872 samples):

• —Train: 2,304 (587 easy / 906 medium / 811 hard)
• —Val: 284 (70/113/101)
• —Test: 284 (68/114/102)
• —100% TD direction, 100% pipe syntax

Hyperparameters

Framework

• —Unsloth for memory-efficient fine-tuning (2× faster, 60% less VRAM)
• —TRL SFTTrainer with UnslothVisionDataCollator
• —transformers 4.57.1, trl 0.22.2

Evaluation

Test set metrics (n=284, V2.2 clean test)

Per-complexity breakdown

Comparison with prior versions

V5 improves edgef1 by **+53%** and labelededge_f1 by +59% over V4 by:

1. 1.Using V2.2 dataset (alt-syntax normalized to pipe syntax)
2. 2.Disabling canonicalize_mermaid (model learns descriptive IDs from raw data)

Intended use

• —Convert flowchart screenshots/diagrams to executable Mermaid code
• —Build AI-assisted flowchart editing tools
• —Document understanding for technical diagrams
• —Pipeline component for diagram-to-text question answering

Limitations

1. 1.TD direction only. Model trained exclusively on top-down flowcharts. Bottom-top, left-right, right-left flows may have lower accuracy.
2. 2.Edge accuracy drops on large flowcharts (≥20 nodes): edge_f1 drops to 0.40 vs 0.90 for small. Use complexity-aware confidence thresholds in production.
3. 3.Synthetic training data. All 2,872 training images are LLM-generated + Mermaid-rendered. Real-world handwritten or scanned flowcharts may show domain gap.
4. 4.English labels only. Training labels are 100% English. Other languages not tested.
5. 5.Mermaid output only. Does not support PlantUML, Graphviz, or other diagram DSLs.
6. 6.No subgraph support. Training data has minimal subgraph usage; complex hierarchical diagrams may flatten.

Training pipeline

Qwen3-VL-8B base model
    ↓
LoRA r=32 α=32 (vision + language unfrozen)
    ↓
Train on V2.2 dataset (2,304 samples, 2 epochs, ~98 min on A100)
    ↓
Merged 16-bit checkpoint (~16 GB)
    ↓
This model

Output format

graph TD
    Start([Start]) --> CheckInput{Is input valid?}
    CheckInput -->|Yes| ProcessData[Process data]
    CheckInput -->|No| ShowError[Show error]
    ProcessData --> End([End])
    ShowError --> End

Format characteristics:

• —graph TD direction (top-down)
• —Descriptive node IDs (Start, CheckInput, not A, B)
• —Pipe-style edge labels: -->|label|
• —Standard shapes: [rect], {rhombus}, (rounded), ([stadium])

Citation

@misc{qwen3vl-flowchart-mermaid-v5,
  title={Qwen3-VL-8B Fine-tuned for Flowchart-to-Mermaid Generation},
  author={Nguyen Hai Dang},
  year={2026},
  howpublished={\url{https://huggingface.co/DangIT02/qwen3vl-flowchart-to-mermaid-v5}},
}

License

Apache 2.0 (inherited from Qwen3-VL base).

Acknowledgments

• —Qwen team for Qwen3-VL-8B base model
• —Unsloth for efficient fine-tuning framework
• —Mermaid.js for the diagram syntax specification