riazmo/design-critic-vlm-3b-lora

Design/UX Critic VLM — Qwen2.5-VL-3B + LoRA (web, v2 / Run 3)

A specialist LoRA adapter that turns Qwen2.5-VL-3B into an on-device UX design critic: given a single UI screenshot, it emits a structured JSON critique (Nielsen heuristics + accessibility, each finding with verdict / severity / evidence / recommendation).

It runs on a laptop, offline, for free. Scoped to web UIs — see Limitations.

Built by a designer as a learning project. The honest write-up (including the two times it failed completely) is the real story — see Provenance below.

TL;DR — what the fine-tune buys you

Same base, same prompt, same images. Measured on 34 hand-built gold screenshots (flag precision/recall on warn/fail findings — not naive agreement, which a "everything's fine" model games).

The untuned base cannot emit a single valid critique. This adapter — ~37M trainable params (~1.2% of the model), 142 MB — is the entire difference.

By slice (read this, not the overall):

Overall recall looks low only because half the gold is native, where the adapter is silent by design. On web it's strong, with 100% precision — zero false alarms.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base = "Qwen/Qwen2.5-VL-3B-Instruct"
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype="auto", device_map={"": "mps"})
model = PeftModel.from_pretrained(model, "riazmo/design-critic-vlm-3b-lora")  # this repo
processor = AutoProcessor.from_pretrained(base)

messages = [{"role": "user", "content": [
    {"type": "image", "image": "screenshot.png"},
    {"type": "text",  "text": INSTRUCTION},   # the v2 rubric instruction (see repo)
]}]
# → generate, then run the JSON sanitizer (drops hallucinated criterion IDs, clamps enums) before parsing.

An inference sanitizer is part of the ship path: it repairs the 3B's occasional malformed output so what you measure is what ships. Without it, valid-JSON rate drops ~20 points.

Limitations (please read)

• —Web only. On native mobile-app screenshots it emits valid JSON but flags nothing — it was not trained on enough bad native examples. Treat any native output as unverified.
• —Single screenshot. No flow/sequence reasoning; no interaction or scroll state.
• —Static, estimated evidence. Contrast etc. are judged by eye, not measured — not a WCAG compliance certifier.
• —Conservative. Misses ~half of real issues on harder web cases. A first-pass assistant, not a sign-off. Small gold set (n=34) — treat percentages as directional.

Method

Knowledge distillation, not prompt tricks. The base 3B rubber-stamped everything (0% recall) because real production screens are ~mostly well-designed. The fix was to relabel the training corpus with a retrieval-augmented stronger teacher (Qwen3-VL-32B + ~6 similar expert critiques; verified 71% recall with no leakage), then distill onto the 3B. QLoRA, rank 16, α 32, vision tower frozen.

Training data & license

Distilled from a corpus of public web UIs, government sites, and design references, plus research datasets. Because that mix includes research-only / portfolio-only sources, this adapter is released `cc-by-nc-4.0` (non-commercial, research/learning use). Do not use commercially. The base model is subject to its own (Qwen) license.

Provenance / write-up

Full engineering story — the failures, the ceiling-check breakthrough, the metric traps — in the accompanying article. This is a learning artifact, published to show the process honestly, not a production model.