DangIT02/qwen3vl-flowchart-to-mermaid_v2

Qwen3-VL-8B — Flowchart to Mermaid (v2)

Vision-language model finetuned to transcribe flowchart diagram images into Mermaid flowchart code.

Given a flowchart image, the model produces valid Mermaid code that reproduces the diagram's nodes, edges, labels, and direction.

Model details

Mermaid canonicalization

Important: This model outputs Mermaid code with canonicalized node IDs — A, B, C, ..., Z, AA, AB, ... in order of first appearance. This is because training data was canonicalized for tool-use compatibility and deterministic output.

Node labels (text inside shapes) and edge labels are preserved exactly.

If you need descriptive IDs (e.g. Start, ProcessPayment), apply a post-processing step to rename nodes based on their labels.

Evaluation (full test set, n=662, canonicalized F1)

Sample counts per bucket: small 208, medium 212, large 242.

Key observations:

• —Medium-complexity flowcharts (10-20 nodes) are the sweet spot — node F1 ≈ 0.80.
• —Large diagrams (20+ nodes) retain strong node accuracy (0.72) but direction detection drops to 90% — the model sometimes defaults to graph TD when the image uses BT or LR.
• —Small diagrams score lower because the model tends to over-generate (hallucinate additional nodes). See Limitations.
• —100% parse success — every generated output is valid Mermaid syntax.

Usage

With transformers

from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image
import torch

model_id = "DangIT02/qwen3vl-flowchart-to-mermaid_v2"
model = AutoModelForImageTextToText.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("flowchart.png").convert("RGB")
if max(image.size) > 1024:
    image.thumbnail((1024, 1024), Image.LANCZOS)

messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": "Convert this flowchart to Mermaid code."},
]}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_tensors="pt", return_dict=True,
).to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=False,
        repetition_penalty=1.15,
    )

mermaid_code = processor.decode(
    out[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)
print(mermaid_code)

With vLLM (faster batched inference)

from vllm import LLM, SamplingParams
from PIL import Image

llm = LLM(
    model="DangIT02/qwen3vl-flowchart-to-mermaid_v2",
    dtype="bfloat16",
    max_model_len=8192,
    limit_mm_per_prompt={"image": 1},
)

sampling = SamplingParams(max_tokens=2048, temperature=0.0, repetition_penalty=1.15)

image = Image.open("flowchart.png").convert("RGB")
prompt = {
    "prompt": "<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Convert this flowchart to Mermaid code.<|im_end|>\n<|im_start|>assistant\n",
    "multi_modal_data": {"image": image},
}
output = llm.generate([prompt], sampling)[0]
print(output.outputs[0].text)

Recommended prompt templates

Any of these works — the model was trained on a variety of phrasings:

• —Convert this flowchart diagram to Mermaid code.
• —Generate the Mermaid code for the provided flowchart.
• —Analyze this flowchart and output the equivalent Mermaid code.
• —What is the Mermaid representation of this flowchart?
• —Transcribe this flowchart into Mermaid syntax.

Limitations

• —Hallucinates on very simple diagrams. On flowcharts with <10 nodes, the model tends to generate 2-3× more nodes than present (mean ratio 3.2×). Likely caused by training-data distribution skew toward medium/complex flowcharts (25.7% easy vs 74.3% medium/hard).
• —Direction detection degrades on large diagrams. For flowcharts with 20+ nodes, direction accuracy drops from 100% to 90%. The model sometimes defaults to graph TD when the true direction is BT or LR. Workaround: include the expected direction in the prompt.
• —Canonicalized node IDs only. The model will not reproduce descriptive IDs from the original diagram. If your downstream tooling requires semantic IDs, post-process the output.
• —English labels only. Dataset is English-only; performance on diagrams with labels in other languages is untested.
• —Single-image input. Model accepts one flowchart image per prompt.
• —Max output ~2048 tokens. Very large flowcharts (>70 nodes) may be truncated.

Training recipe

Trained with Unsloth + TRL's SFTTrainer.

Key hyperparameters:

lora_rank=16, lora_alpha=16, lora_dropout=0.1
finetune_vision_layers=False
finetune_language_layers=True
finetune_attention_modules=True
finetune_mlp_modules=True

num_epochs=2
per_device_train_batch_size=2
gradient_accumulation_steps=8    # effective batch = 16
learning_rate=5e-5
weight_decay=0.001
warmup_ratio=0.05
lr_scheduler_type="cosine"
max_grad_norm=1.0
optim="adamw_8bit"

max_seq_length=4096
seed=3407

Ground-truth Mermaid code was canonicalized before training (node IDs → A, B, C, …) for deterministic output compatible with downstream tool use.

Citation

If you use this model, please cite:

@misc{qwen3vl_flowchart_mermaid_v2,
  title={Qwen3-VL-8B Flowchart-to-Mermaid (v2)},
  author={DangIT02},
  year={2026},
  howpublished={\url{https://huggingface.co/DangIT02/qwen3vl-flowchart-to-mermaid_v2}},
}

Base model: Qwen3-VL.