Tunanzzz/DesignVFR

DesignVFR

Towards Universal Open-Set Visual Font Recognition via Augmented Synthetic Similarity  ·  CVPR 2026 Findings

DesignVFR is the first large-scale dataset for universal open-set Visual Font Recognition (VFR). While prior VFR work is limited to closed-set classification on isolated character-level grayscale images, DesignVFR covers font recognition in real-world universal scenarios — sentences, complex backgrounds, and artistic effects across posters, films, slides and vlogs — and explicitly evaluates the open-set regime where unseen fonts keep being added.

• —📰 Paper: Towards Universal Open-Set Visual Font Recognition via Augmented Synthetic Similarity (CVPR 2026 Findings)
• —🧩 Path scheme: every metadata file uses the ${DATASET_ROOT} placeholder — set it once, and the dataset is portable across machines.

📰 News

• —[2026.06.17] 🎉 DesignVFR is now open-sourced on Hugging Face!
• —[2026.06.17] 🚀 Code release: training, deployment and evaluation pipelines for FontVLM are now public.

✨ Highlights

📦 Subsets

DesignVFR is organised into three complementary subsets:

🗂️ Layout (after python unpack.py)

DesignVFR/
├── synthetic/
│   ├── train/                       1,087 font dirs · 664,375 images
│   ├── id_infer/                    1,068 font dirs ·  55,351 images   (synthetic ID gallery)
│   ├── id_need_infer/               1,068 font dirs ·  55,351 images   (synthetic ID query)
│   ├── ood_infer/                     196 font dirs ·  12,700 images   (synthetic OOD reference imgs)
│   ├── ood_need_infer/                196 font dirs ·  12,700 images   (synthetic OOD query)
│   └── metadata/
│       ├── train.jsonl                       # 664,200 lines · ms-swift conversation format (special <|font|> token)
│       ├── train_sft.jsonl                   # 664,200 lines · plain SFT variant (font name as response)
│       ├── id_infer.json                     #  55,350 records (ID gallery, 1,068 fonts)
│       ├── id_need_infer.json                #  55,350 records (ID query)
│       ├── ood_infer.json                    #  68,050 records — combined ID+OOD gallery used
│       │                                       at OOD evaluation time (1,264 unique fonts)
│       ├── ood_need_infer.json               #  12,700 records (OOD query, 196 fonts)
│       └── font_family_to_index.json         # 1,068-class label map (synthetic training labels)
├── posterreal/
│   ├── images/                     221 font dirs (real poster crops)
│   └── metadata/
│       ├── id.json                 #  9,875 records over  91 fonts (overlap with synthetic train)
│       └── ood.json                # 10,505 records over  82 fonts (unseen during training)
└── videoreal/
    ├── id/                         148 font dirs ·  10,045 frames
    ├── ood/                        170 font dirs ·  12,369 frames
    └── metadata/
        ├── id.json                 # 10,045 records over 130 fonts
        └── ood.json                # 12,369 records over 146 fonts

All paths inside the metadata files use the `${DATASET_ROOT}` placeholder, e.g. "${DATASET_ROOT}/synthetic/train/TsangerYuMo/TsangerYuMo_normal_100_450.png". Set DATASET_ROOT once and every jsonl/json works no matter where you put the dataset on disk.

🚀 Quick Start

1. Download

# requires huggingface_hub>=0.34
hf download Tunanzzz/DesignVFR --repo-type dataset \
    --local-dir ./DesignVFR

2. Unpack the tar shards

The dataset is shipped as ~1 GB tar shards (synthetic/train.part-001.tar, …) to stay friendly to git-LFS. Extract them in place with the bundled unpacker:

cd DesignVFR
python unpack.py

After this step the on-disk layout matches the diagram above. The original .tar shards can be safely deleted.

3. Point your training code at it

export DATASET_ROOT=/abs/path/to/DesignVFR

4. Use it

a. As a datasets.Dataset (raw metadata)

Each metadata JSON is a list of records, each containing the image URL plus a font label:

import json, os
from datasets import Dataset

records = json.load(open('DesignVFR/posterreal/metadata/id.json'))
# Expand ${DATASET_ROOT}
for r in records:
    r['text_img_url'] = os.path.expandvars(r['text_img_url'])

ds = Dataset.from_list(records)
print(ds[0])
# {'font_family': 'Source Han Sans SC',
#  'font-style': 'normal', 'font-weight': '400', 'font-size': '50.6px',
#  'color': '#ffffffFF', 'text': '欢迎咨询',
#  'text_img_url': '/abs/.../posterreal/images/Source Han Sans SC Regular/...png'}

b. With ms-swift (training out of the box)

synthetic/metadata/train.jsonl follows ms-swift's messages format and uses a special <|font|> token as the assistant response:

{"images": ["${DATASET_ROOT}/synthetic/train/TsangerYuMo/TsangerYuMo_normal_100_450.png"],
 "messages": [
   {"role": "user", "content": "<image> What is the font of the text in this image?"},
   {"role": "assistant", "content": "<|font|>"}
 ],
 "label": 0}

ms-swift's preprocessor automatically expands ${DATASET_ROOT} at load time (see swift/llm/dataset/preprocessor/core.py::_cast_mm_data), so launching training is just:

export DATASET_ROOT=/abs/path/to/DesignVFR
swift sft \
    --model Qwen/Qwen2.5-VL-3B-Instruct \
    --dataset $DATASET_ROOT/synthetic/metadata/train.jsonl \
    ...

🔢 Statistics at a glance

About `ood_infer.json`: it has 68,050 records spanning 1,264 fonts, even though synthetic/ood_infer/ on disk only contains the 196 OOD fonts. This is by design — at OOD-evaluation time the gallery must be a superset of the training fonts so that ID classes still serve as distractors. The json therefore concatenates synthetic/id_infer/* (1,068 ID fonts) and synthetic/ood_infer/* (196 OOD fonts) by reference. No data is duplicated on disk.

📑 Metadata schema

synthetic/metadata/{train, train_sft}.jsonl

• —`train.jsonl` — assistant response is the special <|font|> token (used together with a learnable classifier head, see FontVLM).
• —`train_sft.jsonl` — assistant response is the literal font family name (plain SFT). Useful as a baseline / for any model that does not add a special token.

synthetic/metadata/{id,ood}_{infer,need_infer}.json

posterreal/metadata/{id,ood}.json

videoreal/metadata/{id,ood}.json

synthetic/metadata/font_family_to_index.json

{"<font_family_name>": <int_index>} — the canonical 1,068-class label map used during synthetic training.

🧪 Open-set protocol

We split fonts into two disjoint pools:

• —In-Distribution (ID) — the 1,068 fonts seen during synthetic training. *_id query splits are evaluated by top-k accuracy against an ID gallery of size 1,068.
• —Out-of-Distribution (OOD) — fonts never seen during training (177 unique families across all OOD splits). *_ood query splits target the open-set capability of the recogniser; their gallery is the combined ID+OOD reference (synthetic/metadata/ood_infer.json, 1,264 fonts).

The accompanying paper recommends two evaluation modes:

1. 1.Classification mode — direct softmax over the 1,068 ID classes. Applicable only on ID splits.
2. 2.Similarity mode — extract a query feature, then match it via cosine similarity against the synthetic reference gallery. This naturally extends to OOD fonts at inference time, without any retraining.

Reference implementations of both modes will be released alongside the FontVLM codebase (coming soon).

⚖️ License & responsible use

• —The dataset is released under the Apache 2.0 license, for research purposes only.
• —The font files themselves are not redistributed — only rendered images are. Some fonts shipped in DesignVFR carry restrictive commercial licenses (e.g. fonts whose names contain "Non-Commercial Use"); please consult the original font foundries before any commercial application.
• —Real-world poster / video frames were collected from public sources for academic study only. If you are a copyright holder and would like a sample removed, please open a thread in the Community tab of this dataset and we will take it down.

📚 Citation

@InProceedings{Zhou_2026_CVPR,
    author    = {Zhou, Peicheng and Fang, Shancheng and Jin, Chenhui and Pu, Bowei and Xie, Hongtao},
    title     = {Towards Universal Open-Set Visual Font Recognition Via Augmented Synthetic Similarity},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings},
    month     = {June},
    year      = {2026},
    pages     = {6799-6808}
}

🙏 Acknowledgement

We thank the open-source projects ms-swift, Qwen2.5-VL, LLaVA-OneVision, and PaddleOCR, on top of which DesignVFR was built.