Modelpublic

shibuina/drawvla-verify-smolvlm-500m-lora

sourceHugging Faceapache-2.0updated 18d agoView on Hugging Face

0likes14downloads

Model Card

drawvla-verify-smolvlm-500m-lora

LoRA adapter for HuggingFaceTB/SmolVLM-500M-Instruct trained as a sketch-prompt validator (DrawVLA-Verify): given a robot-scene image with a drawn circle (which object) and arrow (where), plus a name-free caption such as "put this there", decide whether the instruction is right (correct, consistent, sufficient) or wrong.

Trained on shibuina/drawvla-prompt-validation (LIBERO-grounded synthetic prompts; 14 corruption modes + grounding configurations). LoRA r=16, alpha=32, dropout 0.05, all attention/MLP projections, 3 epochs, lr 1e-4 cosine, 3% warmup, seed 17.

Results (held-out physical-free test split, 3,966 rows; positive class = valid)

split	accuracy	balanced acc	macro-F1	MCC	AUROC
synthetic test	0.895	0.836	0.855	0.716	0.949

Recall on invalid prompts 0.959, on valid prompts 0.714. A geometry-only classifier on the same test split reaches 0.932 accuracy, so the synthetic benchmark is largely separable from sketch coordinates; treat the human-sketch numbers as the deployment estimate.

Usage

The adapter must be scored the way it was trained: the user turn is the image plus "<VERDICT_INSTRUCTION>\n\nCaption: <caption>", and the verdict is chosen by the label-span log-likelihood of verdict: right vs verdict: wrong (see training/vlm_training.py and training/train_local.py --evaluate-adapter in the DrawVLA repository). Prompting it differently (e.g. a system prompt + bare next-token scoring) roughly halves the margins.

python

from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
base = "HuggingFaceTB/SmolVLM-500M-Instruct"
model = PeftModel.from_pretrained(AutoModelForImageTextToText.from_pretrained(base, dtype="bfloat16"), "shibuina/drawvla-verify-smolvlm-500m-lora")
processor = AutoProcessor.from_pretrained(base)

Files: adapterconfig.json, adaptermodel.safetensors, chattemplate.jinja, processorconfig.json, runconfig.json, testgenerationmetrics.json, tokenizer.json, tokenizerconfig.json, train_metrics.json.