OpenMOSS-Team/MOSS-VL-Instruct-0408

1---2title: MOSS-VL-Instruct-04083date: '2026-04-08T00:00:00.000Z'4category: Multimodal-LLM5status: SFT6language:7- en8- zh9library_name: transformers10pipeline_tag: video-text-to-text11license: apache-2.012base_model: fnlp-vision/MOSS-VL-Base-040813tags:14- SFT15- Video-Understanding16- Image-Understanding17- MOSS-VL18- OpenMOSS19- multimodal20- video21- vision-language22---23 24<p align="center">25   <img src="assets/logo.png" width="320"/>26</p>27 28# MOSS-VL-Instruct-040829 30MOSS-VL is an open vision-language model family from OpenMOSS, supporting image understanding, long-video understanding, and realtime streaming interaction.31 32**Technical Report**: [https://arxiv.org/pdf/2608.15045](https://arxiv.org/pdf/2608.15045)33 34## 📌 Introduction35 36MOSS-VL-Instruct-0408 is the instruction-tuned checkpoint of the MOSS-VL series, part of the OpenMOSS ecosystem dedicated to advancing visual understanding.37 38Built on top of MOSS-VL-Base-0408 through supervised fine-tuning (SFT), this checkpoint is designed as a high-performance offline multimodal engine. It delivers strong, well-rounded performance across the full spectrum of vision-language tasks — including image understanding, OCR, document parsing, visual reasoning, and instruction following — and is particularly outstanding at video understanding, from long-form comprehension to fine-grained temporal reasoning and action recognition.39 40### ✨ Highlights41 42- 🎬 **Outstanding Video Understanding** — A core strength of MOSS-VL. The model excels at long-form video comprehension, temporal reasoning, action recognition, and second-level event localization, delivering top-tier results on benchmarks such as VideoMME, and MLVU.43- 🖼️ **Strong General Multimodal Perception** — Robust image understanding, fine-grained object recognition, OCR, and document parsing.44- 💬 **Reliable Instruction Following** — Substantially improved alignment with user intent through supervised fine-tuning on diverse multimodal instruction data.45 46 47---48 49## 🏗 Model Architecture50 51**MOSS-VL-Instruct-0408** adopts a cross-attention-based architecture that decouples visual encoding from cognitive reasoning. This design drives latency down to the **millisecond level**, enabling instantaneous responses to dynamic video streams. Natively supporting **interleaved modalities**, it processes complex sequences of images and videos within a unified pipeline — eliminating the need for heavy pre-processing.52 53<p align="center">54    <img src="assets/structure.png" alt="MOSS-VL Architecture" width="90%"/>55</p>56 57## 🧩 Absolute Timestamps58 59To ensure the model accurately perceives the pacing and duration of events, **MOSS-VL-Instruct-0408** injects **absolute timestamps** alongside each sampled frame, grounding the reasoning process in a **precise temporal reference**.60 61<p align="center">62    <img src="assets/timestamp_input.svg" alt="Timestamped Sequence Input Illustration" width="90%"/>63</p>64 65## 🧬 Cross-attention RoPE (XRoPE)66 67MOSS-VL utilizes Cross-attention Rotary Position Embedding (XRoPE), tailored to its cross-attention based vision–language architecture. This mechanism maps text tokens and video patches into a unified 3D coordinate space defined by Time (t), Height (h), and Width (w).68 69<p align="center">70    <img src="assets/3d-rope.png" alt="MOSS-VL mRoPE Architecture Illustration" width="80%"/>71</p>72 73## 📊 Model Performance74 75We conducted a comprehensive evaluation of **MOSS-VL-Instruct-0408** across four key dimensions: Multimodal Perception, Multimodal Reasoning, Document/OCR, and Video Understanding. The results demonstrate that MOSS-VL achieves outstanding performance, particularly excelling in **general multimodal perception** and **complex video analysis**.76 77### 🌟 Key Highlights78 79*   **🚀 Leading Video Intelligence**: MOSS-VL achieves a score of **65.8** in Video Understanding, significantly outperforming Qwen3-VL (+2pts). It shows exceptional temporal consistency and action recognition capabilities across benchmarks like `VideoMME`, `MLVU`, `EgoSchema`, and `VSI-bench` (where it outperforms **Qwen3-VL-8B-Instruct** by **8.3 points**).80*   **👁️ Outstanding Multimodal Perception**: MOSS-VL delivers excellent general image-text understanding, shining in fine-grained object recognition and spatial reasoning on benchmarks like `BLINK` and `MMBench`.81*   **🧠 Robust Multimodal Reasoning**: MOSS-VL demonstrates solid logical inference, staying highly competitive with the latest Qwen series on challenging reasoning suites.82*   **📄 Reliable Document Understanding**: While the model is primarily optimized for general perception, MOSS-VL still delivers **83.9** on OCR and document analysis, ensuring dependable extraction of text and structured information.83 84 85<p align="center">86    <img src="assets/MOSS-VL-benchmark.png" alt="MOSS-VL Benchmark Results" width="100%"/>87</p>88 89## 🚀 Quickstart90### 🛠️ Installation91 92```bash93conda create -n moss_vl python=3.12 pip -y94conda activate moss_vl95pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt96```97 98### 🏃 Run Inference99 100 101<details>102<summary><strong>Single-image offline inference (Python)</strong></summary>103 104<br>105 106```python107import torch108from transformers import AutoModelForCausalLM, AutoProcessor109 110checkpoint = "path/to/checkpoint"111image_path = "data/example_image.jpg"112prompt = "Describe this image."113 114 115def load_model(checkpoint: str):116    processor = AutoProcessor.from_pretrained(117        checkpoint,118        trust_remote_code=True,119        frame_extract_num_threads=1,120    )121    model = AutoModelForCausalLM.from_pretrained(122        checkpoint,123        trust_remote_code=True,124        device_map="auto",125        torch_dtype=torch.bfloat16,126        attn_implementation="flash_attention_2",127    )128    return model, processor129 130 131model, processor = load_model(checkpoint)132 133text = model.offline_image_generate(134    processor,135    prompt=prompt,136    image=image_path,137    shortest_edge=4096,138    longest_edge=16777216,139    multi_image_max_pixels=201326592,140    patch_size=16,141    temporal_patch_size=1,142    merge_size=2,143    image_mean=[0.5, 0.5, 0.5],144    image_std=[0.5, 0.5, 0.5],145    max_new_tokens=256,146    temperature=1.0,147    top_k=50,148    top_p=1.0,149    repetition_penalty=1.0,150    do_sample=False,151    vision_chunked_length=64,152)153 154print(text)155```156 157</details>158 159<details>160<summary><strong>Single-video offline inference (Python)</strong></summary>161 162<br>163 164```python165import torch166from transformers import AutoModelForCausalLM, AutoProcessor167 168checkpoint = "path/to/checkpoint"169video_path = "data/example_video.mp4"170prompt = "Describe this video."171 172 173def load_model(checkpoint: str):174    processor = AutoProcessor.from_pretrained(175        checkpoint,176        trust_remote_code=True,177        frame_extract_num_threads=1,178    )179    model = AutoModelForCausalLM.from_pretrained(180        checkpoint,181        trust_remote_code=True,182        device_map="auto",183        torch_dtype=torch.bfloat16,184        attn_implementation="flash_attention_2",185    )186    return model, processor187 188 189model, processor = load_model(checkpoint)190 191text = model.offline_video_generate(192    processor,193    prompt=prompt,194    video=video_path,195    shortest_edge=4096,196    longest_edge=16777216,197    video_max_pixels=201326592,198    patch_size=16,199    temporal_patch_size=1,200    merge_size=2,201    video_fps=1.0,202    min_frames=1,203    max_frames=256,204    num_extract_threads=4,205    image_mean=[0.5, 0.5, 0.5],206    image_std=[0.5, 0.5, 0.5],207    max_new_tokens=256,208    temperature=1.0,209    top_k=50,210    top_p=1.0,211    repetition_penalty=1.0,212    do_sample=False,213    vision_chunked_length=64,214)215 216print(text)217```218 219</details>220 221<details>222<summary><strong>Batched offline inference (Python)</strong></summary>223 224<br>225 226```python227import torch228from transformers import AutoModelForCausalLM, AutoProcessor229 230checkpoint = "path/to/checkpoint"231processor = AutoProcessor.from_pretrained(232    checkpoint,233    trust_remote_code=True,234    frame_extract_num_threads=1,235)236model = AutoModelForCausalLM.from_pretrained(237    checkpoint,238    trust_remote_code=True,239    device_map="auto",240    torch_dtype=torch.bfloat16,241    attn_implementation="flash_attention_2",242)243 244queries = [245    {246        "prompt": "Describe sample A.",247        "images": [],248        "videos": ["data/sample_a.mp4"],249        "media_kwargs": {"video_fps": 1.0, "min_frames": 8, "max_frames": 256},250        "generate_kwargs": {251            "temperature": 1.0,252            "top_k": 50,253            "top_p": 1.0,254            "max_new_tokens": 256,255            "repetition_penalty": 1.0,256            "do_sample": False,257        },258    },259    {260        "prompt": "Describe sample B.",261        "images": [],262        "videos": ["data/sample_b.mp4"],263        "media_kwargs": {"video_fps": 1.0, "min_frames": 8, "max_frames": 256},264        "generate_kwargs": {265            "temperature": 1.0,266            "top_k": 50,267            "top_p": 1.0,268            "max_new_tokens": 256,269            "repetition_penalty": 1.0,270            "do_sample": False,271        },272    },273]274 275with torch.no_grad():276    result = model.offline_batch_generate(processor, queries, vision_chunked_length=64)277 278texts = [item["text"] for item in result["results"]]279```280 281</details>282 283## 🚧 Limitations and Future Work284 285MOSS-VL-Instruct-0408 represents an early milestone in the MOSS-VL roadmap, and we're actively working on several directions to push it further:286 287- 🧮 **Math & Code Reasoning** — While the current checkpoint already exhibits great general reasoning, we plan to substantially strengthen its mathematical reasoning and code reasoning capabilities, especially in multimodal contexts.288- 🎯 **RL Post-Training** — We are working on a reinforcement learning post-training stage to further align the model with human preferences and to unlock stronger multi-step reasoning behaviors on top of the SFT foundation.289 290 291> [!NOTE]292> We welcome community feedback and contributions on any of these directions.293 294 295 296## 📜 Citation297```bibtex298@misc{mossvl,299  title         = {MOSS-VL Technical Report},300  author        = {Wang, Pengyu and Tan, Chenkun and Zhou, Shaojun and Zhou, Qirui and Chen, Yanxin and He, Xingyang and Zeng, Huazheng and Cheng, Jijun and Wang, Chenghao and Qian, Xiaomeng and Wang, Pengfei and Huang, Zhan and Gao, Shanqing and Huang, Wei and Cao, Longjun and Ran, Wu and Liu, Jie and Zhu, Changtai and Wang, Hongkai and Tian, Yixian and Liu, Chenghao and Ye, Zhen and Wang, Xinghao and Jiang, Botian and Feng, Guoguo and Fei, Zhaoye and Li, Ruixiao and Chen, Mingshu and Gao, Yang and Cheng, Qinyuan and Li, Shimin and Qiu, Xipeng},301  year          = {2026},302  eprint        = {2608.15045},303  archivePrefix = {arXiv},304  primaryClass  = {cs.CV},305  url           = {https://arxiv.org/abs/2608.15045}306}307 308@misc{mossvideopreview,309  title         = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},310  author        = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu},311  year          = {2026},312  eprint        = {2606.07639},313  archivePrefix = {arXiv},314  primaryClass  = {cs.CV},315  url           = {https://arxiv.org/abs/2606.07639}316}317```