hsiangfu/ReVision-250M-256-16-baseline

Model Card for ReVision-250M-256-16-baseline

This repository contains ReVision-250M-256-16-baseline, a compact vision-language model (VLM) designed for Visual Instruction Rewriting. The model rewrites multimodal task-oriented instructions into text-only commands, enabling privacy-preserving on-device AI by eliminating the need to process images in the cloud.

Key Features

• —Lightweight (250M parameters): Designed for on-device deployment with efficient inference.
• —Privacy-Preserving: Converts multimodal inputs into structured text, reducing reliance on cloud-based processing.
• —Fine-Tuned for Instruction Rewriting: Trained on a dataset of 39,000 examples spanning 14 task-oriented domains.
• —Compact Yet Effective: Outperforms larger models like PaliGemma-v2 (10B) and QwenVL-7B in instruction rewriting tasks.

Model Architecture

• —Vision Encoder: google/siglip-base-patch16-256 (processes 256×256 images).
• —Language Model: OuteAI/Lite-Mistral-150M-v2-Instruct (instruction-tuned).
• —Multimodal Fusion: Uses a linear projector to align vision and language embeddings.
• —Training Dataset: Pretrained on image captioning datasets (e.g., LLaVA-CC3M, LLaVA-Pretrain) and fine-tuned on the Visual Instruction Rewriting dataset.

Performance

How to Use

Install Dependencies

pip install torch transformers torchvision  

Load the Model

from transformers import AutoProcessor, AutoModelForSeq2SeqLM
import torch
from PIL import Image

# Load model and processor
model_name = "hsiangfu/ReVision-250M-256-16-baseline"
processor = AutoProcessor.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Prepare inputs (image + instruction)
image = Image.open("example.jpg")
instruction = "Call this number."

inputs = processor(images=image, text=instruction, return_tensors="pt")
outputs = model.generate(**inputs)

# Decode rewritten instruction
rewritten_instruction = processor.batch_decode(outputs, skip_special_tokens=True)[0]
print("Rewritten Instruction:", rewritten_instruction)

Dataset

The model was fine-tuned on the ReVision Multimodal Query Rewrites Dataset, a collection of 39,023 ⟨image, original instruction, rewritten instruction⟩ triplets covering:

• —Books: "Who wrote this book" → "Who wrote 'The Silent Patient'?"
• —Business Cards: "Call this number." → "Call 512-555-1234."
• —Flyers & Signboards: "Add this event to my calendar." → "Add 'Tech Conference' on May 5 at 2 PM to my calendar."
• —Landmarks: "Who made this?" → "Who made the Statue of Liberty?"
• —Products: "What brand is this product?" → "What brand made 'Mismatched Sandwich Cremes'?"
• —CD covers: "Who made this CD?" → "Who made 'Future'?"
• —Paintings: "Who is this painting by?" → "Who made the painting 'Mona Lisa'?"

Link: https://huggingface.co/datasets/hsiangfu/multimodalqueryrewrites

Applications

• —AR/VR Assistants (e.g., Apple Vision Pro, Meta Ray-Ban Glasses)
• —Smartphones & Wearables (on-device AI assistants)
• —Accessibility & Assistive AI (for users with visual impairments)

Citation

Acknowledgments

Developed by researchers at UT Austin and Yale University. Model and dataset are available for academic use.