haofue2i1z3/Mini-VLM-Qwen3-1.7B-SigLIP

Mini-VLM (Qwen3-1.7B + SigLIP)

A 2.2B vision-language model trained with TinyLLaVA: SigLIP-SO400M/14 vision encoder, a 2-layer MLP connector, and Qwen3-1.7B in non-thinking mode. Trained on a single RTX 5090.

Code: https://github.com/jackychris/mini-vlm

Training

• —Stage 1: 150K LLaVA-Pretrain captions, connector only (6.6M parameters).
• —Stage 2: 150K samples from LLaVA-1.5 mix665k (COCO / GQA / ShareGPT) plus 15K Chinese samples from BUAADreamer/llava-en-zh-300k. Qwen3 and the connector are trained (1.73B parameters); the vision encoder stays frozen.
• —Both stages: 1 epoch, cosine schedule with 3% warmup, bf16. Stage 1 uses lr 1e-3 and batch size 256; Stage 2 uses lr 2e-5 and batch size 128.

Results

Greedy decoding. Only the leading yes/no or option letter is parsed, and unparseable answers count as wrong.

Limitations

• —Strong bias toward answering "yes" on POPE; no recall is 43.1% on the adversarial split.
• —One 384×384 image per input, so small text and dense documents are not read reliably.
• —No safety alignment.

Usage

The checkpoint uses TinyLLaVA's model class, so it needs the TinyLLaVA code with the Qwen3 patches from the GitHub repo instead of the Transformers Auto* classes. Tested with Python 3.10, PyTorch 2.12.1+cu130, and Transformers 4.51.3.

git clone https://github.com/jackychris/mini-vlm mini-vlm && cd mini-vlm
bash scripts/bootstrap_server.sh
hf download haofue2i1z3/Mini-VLM-Qwen3-1.7B-SigLIP --local-dir checkpoints/mini-vlm

from pathlib import Path
from PIL import Image
from scripts.run_stage2_evaluation import TinyLlavaEvaluator

model = TinyLlavaEvaluator(Path("checkpoints/mini-vlm"))
image = Image.open("demo.jpg").convert("RGB")
print(model.generate(image, "Describe this image.", max_new_tokens=128))

Notes

• —The weights are the evaluated checkpoint, unchanged. In config.json, local paths from the training machine were replaced with Qwen/Qwen3-1.7B, google/siglip-so400m-patch14-384, and this repo so the model loads elsewhere.
• —The training data (LLaVA-Pretrain, COCO, GQA, ShareGPT, llava-en-zh-300k) has its own terms. Use the model for research and learning.