OpenMOSS-Team/MOSS-VL-Realtime-NF4

1---2license: apache-2.03language:4- en5- zh6library_name: transformers7pipeline_tag: video-text-to-text8base_model: OpenMOSS-Team/MOSS-VL-Realtime9tags:10- MOSS-VL11- realtime12- streaming13- video-understanding14- bitsandbytes15- NF416- quantized17- custom_code18---19 20<p align="center">21  <img src="assets/logo.png" width="300" alt="MOSS-VL"/>22</p>23 24<p align="center">25  English | <a href="https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime-NF4/blob/main/README_zh.md">中文</a>26</p>27 28# MOSS-VL-Realtime W4A16 NF4 + KV8 HQQ29 30MOSS-VL is an open vision-language model family from OpenMOSS, supporting image understanding, long-video understanding, and realtime streaming interaction. This repository provides the W4A16 NF4 + KV8-quantized checkpoint of MOSS-VL-Realtime.31 32**Technical Report**: [https://arxiv.org/pdf/2608.15045](https://arxiv.org/pdf/2608.15045)33 34This is the 24 GiB quantized release of MOSS-VL-Realtime. It keeps the original35timestamp-aware streaming interface and can also use the offline image/video36helpers from the standard checkpoint.37 38## Quantization profile39 40| Component | Format |41| --- | --- |42| Most language layers | bitsandbytes NF4 4-bit weights with double quantization |43| First/last language layers and multimodal modules | BF16 |44| Activations and compute | BF16 |45| Transformers KV cache | HQQ INT8 |46| Attention backend | FlashAttention 2 |47 48The checkpoint carries its bitsandbytes configuration, HQQ cache configuration,49and MOSS-VL remote modeling code. Load the directory directly; do not add a50second runtime quantization configuration.51 52## Quantization benchmark53 54Across the selected benchmarks, the quantized models remain close to their55non-quantized BF16 counterparts, showing that overall model quality is largely56preserved after quantization.57 58<p align="center">59  <img src="assets/mossvl_quantization_benchmark_comparison_en_4k.png" alt="MOSS-VL quantization benchmark comparison" width="100%"/>60</p>61 62## Hardware requirements63 64The model is designed to run on a single NVIDIA GPU with 24 GB of VRAM. Use65FlashAttention 2 and `frame_queue_size=1` for the 24 GB realtime profile.66 67## Environment68 69### Installation70 71Use the standard MOSS-VL repository requirements, then add the two quantization72backends required by this checkpoint:73 74```bash75git clone https://github.com/OpenMOSS/MOSS-VL.git76cd MOSS-VL77 78conda create -n moss_vl_quant python=3.12 pip -y79conda activate moss_vl_quant80pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt81pip install -i https://pypi.org/simple \82  bitsandbytes==0.49.2 \83  hqq==0.2.8.post184python -m pip check85```86 87The standard release environment uses the following core stack:88 89| Package | Version |90| --- | --- |91| Python | 3.12 |92| PyTorch | 2.8.0 + CUDA 12.8 |93| Transformers | 4.57.1 |94| Accelerate | 1.12.0 |95| FlashAttention | 2.8.1 |96| bitsandbytes | 0.49.2 |97| HQQ | 0.2.8.post1 |98 99Video decoding also requires FFmpeg to be available in `PATH`.100 101## Load the model102 103Keep `attn_implementation` set to `flash_attention_2` for the 24 GB profile.104 105```python106import torch107from transformers import AutoModelForCausalLM, AutoProcessor108 109checkpoint = "/path/to/mossvl_streaming_w4a16_nf4_keep_first4_last4_kv8_hqq"110 111processor = AutoProcessor.from_pretrained(112    checkpoint,113    trust_remote_code=True,114    frame_extract_num_threads=1,115)116model = AutoModelForCausalLM.from_pretrained(117    checkpoint,118    trust_remote_code=True,119    device_map="auto",120    torch_dtype=torch.bfloat16,121    attn_implementation="flash_attention_2",122)123model.eval()124```125 126`generation_config.json` automatically enables the HQQ INT8 KV cache. Do not127override it with a BF16/dynamic cache when using the 24 GB profile.128 129## Realtime inference130 131The application supplies PIL-compatible frames with non-decreasing timestamps.132Use `frame_queue_size=1` for the 24 GB realtime profile.133 134```python135import time136from PIL import Image137 138session = model.create_realtime_session(139    processor,140    initial_prompt=(141        "Describe important changes in the video as they happen. "142        "Stay silent when there is no meaningful update."143    ),144    frame_queue_size=1,145    max_tokens_per_turn=12,146    max_new_tokens=4096,147    do_sample=False,148)149 150frame_paths = [151    "data/frame_0001.jpg",152    "data/frame_0002.jpg",153    "data/frame_0003.jpg",154]155 156try:157    session.start()158    for index, frame_path in enumerate(frame_paths):159        image = Image.open(frame_path).convert("RGB")160        session.push_frame(image, timestamp=float(index))161 162        while True:163            chunk = session.poll_output(timeout=0.0)164            if chunk is None:165                break166            print(chunk, end="", flush=True)167 168        time.sleep(1.0)169 170    session.push_prompt("What changed in the latest frames?")171    deadline = time.monotonic() + 5.0172    while time.monotonic() < deadline:173        chunk = session.poll_output(timeout=0.1)174        if chunk is not None:175            print(chunk, end="", flush=True)176finally:177    session.close()178```179 180One model instance supports one active realtime session. The model may emit181control tokens such as `<|silence|>`, `<|round_start|>`, and `<|round_end|>`;182applications should filter or render them according to their protocol.183 184## Offline video inference185 186```python187text = model.offline_video_generate(188    processor,189    prompt="Describe this video.",190    video="data/example_video.mp4",191    shortest_edge=4096,192    longest_edge=16777216,193    video_max_pixels=201326592,194    patch_size=16,195    temporal_patch_size=1,196    merge_size=2,197    video_fps=1.0,198    min_frames=1,199    max_frames=256,200    num_extract_threads=4,201    image_mean=[0.5, 0.5, 0.5],202    image_std=[0.5, 0.5, 0.5],203    max_new_tokens=256,204    temperature=1.0,205    top_k=50,206    top_p=1.0,207    repetition_penalty=1.0,208    do_sample=False,209    vision_chunked_length=64,210)211print(text)212```213 214## Configuration files215 216- `config.json`: model and NF4 weight configuration.217- `generation_config.json`: HQQ KV8 configuration.218- `modeling_moss_vl.py`: checkpoint-local MOSS-VL and QuantizedCache code.219 220## Citation221 222```bibtex223@misc{mossvl,224  title         = {MOSS-VL Technical Report},225  author        = {Wang, Pengyu and Tan, Chenkun and Zhou, Shaojun and Zhou, Qirui and Chen, Yanxin and He, Xingyang and Zeng, Huazheng and Cheng, Jijun and Wang, Chenghao and Qian, Xiaomeng and Wang, Pengfei and Huang, Zhan and Gao, Shanqing and Huang, Wei and Cao, Longjun and Ran, Wu and Liu, Jie and Zhu, Changtai and Wang, Hongkai and Tian, Yixian and Liu, Chenghao and Ye, Zhen and Wang, Xinghao and Jiang, Botian and Feng, Guoguo and Fei, Zhaoye and Li, Ruixiao and Chen, Mingshu and Gao, Yang and Cheng, Qinyuan and Li, Shimin and Qiu, Xipeng},226  year          = {2026},227  eprint        = {2608.15045},228  archivePrefix = {arXiv},229  primaryClass  = {cs.CV},230  url           = {https://arxiv.org/abs/2608.15045}231}232 233@misc{mossvideopreview,234  title         = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},235  author        = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu},236  year          = {2026},237  eprint        = {2606.07639},238  archivePrefix = {arXiv},239  primaryClass  = {cs.CV},240  url           = {https://arxiv.org/abs/2606.07639}241}242```243