microsoft/mage-vl-demo

Mage-VL — codec-native streaming multimodal model

Demo of `microsoft/Mage-VL`, a 4B codec-native vision-language model (Mage-ViT encoder trained from scratch + Qwen3-4B-Instruct-2507 decoder).

Instead of decoding video into uniformly sampled frames and pushing a dense grid of patch tokens through a ViT, Mage-VL follows the structure of a video codec: it keeps every anchor (I) frame patch and only the predicted (P) frame patches where the codec spends bits — the regions carrying real motion and new detail. Those surviving patches are packed into canvases, cutting visual tokens by >75%.

Tabs:

• —Image — single-image Q&A.
• —Video — video Q&A, switchable between the codec-native (HEVC) tokenizer and classic uniform frame sampling, showing the canvases / frames actually fed to the model plus the visual-token count for each path.
• —Proactive streaming — the StreamMind cognition gate scores each rolling codec segment and the model only speaks when a response-worthy event completes.

Preprocessing follows the reference inference.py / inference_streaming.py from the model repo and the Mage GitHub repo.