chris0809/tinyLLM-0.51B-VLM

tinyLLM VLM 0.51B

<!-- tinyllm-showcase --> 中文项目文档 · English project overview · Live demo


The local demo switches to the VLM checkpoint, encodes five image views, and streams the checked output. <!-- /tinyllm-showcase -->

tinyLLM is a custom 0.51B decoder-only model trained from scratch for Chinese and English. Loading requires trust_remote_code=True because the architecture is implemented in this repository.

This release is the step 18000 VLM checkpoint. It combines the language model with a 4-layer Q-Former, projector and rank-128 visual LoRA. The visual continual-SFT stage used 201,748 image examples plus 20% bilingual text replay.

Load

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "chris0809/tinyLLM-0.51B-VLM"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
)
model.load_lora_pretrained(model_id, subfolder="adapter")

Thinking mode

The bundled tokenizer also accepts enable_thinking=True in apply_chat_template, mainly for text-only mathematics or multi-step reasoning. Keep it off for ordinary image description; the GitHub demo uses a separate conservative visual system prompt to reduce hallucination.

Vision encoder

The frozen vision tower is OpenGVLab/InternViT-300M-448px-V2_5 and is loaded separately. This repository contains the language model, Q-Former, projector and the visual LoRA adapter. The model forward accepts precomputed vision_feats, vision_mask, global_pos and global_off tensors. The five-view image preprocessing and end-to-end chat path are in the GitHub project. This repository does not claim generic AutoProcessor support.

Source

Training code and full project documentation: https://github.com/Huanz86251/tinyLLM