shi-labs/slowfast-video-mllm-qwen2-7b-convnext-576-frame64-s1t4

# Slow-Fast Architecture for Video Multi-Modal Large Language Models (Qwen2-7B, 64 Frames)

This repository contains the **Slow-Fast Video MLLM (Qwen2-7B, ConvNeXt-576, 64 frames, stride 1/4)** model, presented in the paper [Slow-Fast Architecture for Video Multi-Modal Large Language Models](https://huggingface.co/papers/2504.01328).

[Code Repository](https://github.com/SHI-Labs/Slow-Fast-Video-Multimodal-LLM) | [HuggingFace Collection](https://huggingface.co/collections/shi-labs/slow-fast-video-mllm-67ef347a28772734c15a78b5)

## Model Description

This model introduces a novel slow-fast architecture to address the challenge of balancing temporal resolution and spatial detail in video-based multi-modal large language models (MLLMs) under limited compute budgets. Existing methods often compress video representations irreversibly, losing detail.

Inspired by how humans first skim a video before focusing on relevant parts, the slow-fast design employs a dual-token strategy:
1.  **"Fast" visual tokens:** A compact set of compressed video features fed into the LLM (Qwen2-7B-Instruct) alongside text embeddings for a quick overview.
2.  **"Slow" visual tokens:** Uncompressed video features cross-attended by text embeddings via specially designed hybrid decoder layers, enabling instruction-aware extraction of relevant visual details with linear complexity.

This approach allows processing more input frames (e.g., 64 frames for this checkpoint) while preserving spatial details, leading to significant performance improvements on video understanding benchmarks compared to self-attention-only baselines. This checkpoint uses a Qwen2-7B-Instruct base LLM and a ConvNeXt-576 vision tower.

<div align="center">
<img src="https://huggingface.co/shi-labs/slowfast-video-mllm-qwen2-7b-convnext-576-frame64-s1t4/resolve/main/assets/images/fig-teaser.png" width="45%">
</div>

## Usage

**Note:** This model relies on custom code integrated within the `transformers` library (`LlavaQwenSlowFastForCausalLM`). Ensure you have the necessary packages installed from the [official repository](https://github.com/SHI-Labs/Slow-Fast-Video-Multimodal-LLM) or use `trust_remote_code=True` when loading the model.

First, clone the repository and install requirements if running locally:

git clone https://github.com/SHI-Labs/Slow-Fast-Video-Multimodal-LLM.git cd Slow-Fast-Video-Multimodal-LLM pip install --upgrade pip pip install -r requirements.txt

Add the cloned repo path to your PYTHONPATH or install it

Then, use the following Python script:

import torch import os import numpy as np from decord import VideoReader, cpu import requests # Required to download video

Make sure the necessary llava modules are importable

If not installed from the repo, trustremotecode=True handles this

from llava.constants import IMAGETOKENINDEX, DEFAULTIMAGETOKEN, DEFAULTIMSTARTTOKEN, DEFAULTIMENDTOKEN from llava.conversation import convtemplates from llava.model.builder import loadpretrainedmodel from llava.mmutils import tokenizerimagetoken, getmodelnamefrompath from llava.utils import disabletorchinit

def loadvideo(videopath, maxframesnum): """Helper function to load video frames.""" vr = VideoReader(videopath, numthreads=4) total_frames = len(vr)

# Ensure sparse sampling doesn't lead to fewer frames than requested if totalframes >= maxframesnum: # Uniformly sample frames across the video uniformsampledframes = np.linspace(0, totalframes - 1, maxframesnum, dtype=int) frameidx = uniformsampledframes.tolist() else: # If video is shorter than maxframesnum, sample all frames and repeat the last frameidx = list(range(totalframes)) frameidx.extend([totalframes - 1] * (maxframesnum - totalframes))

try: spareframes = vr.getbatch(frameidx).asnumpy() except Exception as e: print(f"Error loading video frames: {e}") # Fallback or error handling: return None or raise exception # Example: return a black frame tensor of the expected shape # This part depends on how imageprocessor handles None or errors # For now, re-raising the exception might be best raise e

return spare_frames

Model configuration

modelpath = "shi-labs/slowfast-video-mllm-qwen2-7b-convnext-576-frame64-s1t4" videourl = "https://huggingface.co/shi-labs/slowfast-video-mllm-qwen2-7b-convnext-576-frame64-s1t4/resolve/main/assets/catinterrupt.mp4" videolocalpath = "catinterrupt.mp4" question = "Please describe this video in detail." max_frames = 64 # This checkpoint was trained with 64 frames

Download the video if it doesn't exist

if not os.path.exists(videolocalpath): print(f"Downloading video from {videourl}...") response = requests.get(videourl, stream=True) response.raiseforstatus() # Raise an exception for bad status codes with open(videolocalpath, "wb") as f: for chunk in response.itercontent(chunksize=8192): f.write(chunk) print("Download complete.")

Load the model and processor

disabletorchinit() modelname = getmodelnamefrompath(modelpath)

Use trustremotecode=True to load the custom architecture

tokenizer, model, imageprocessor, contextlen = loadpretrainedmodel( modelpath, None, modelname, useflashattn=True, # Use Flash Attention if available devicemap="auto", # Automatically distribute model across GPUs/CPU torchdtype=torch.bfloat16, # Use bfloat16 for efficiency trustremotecode=True )

Prepare the prompt

if model.config.mmuseimstartend: prompt = DEFAULTIMSTARTTOKEN + DEFAULTIMAGETOKEN + DEFAULTIMENDTOKEN + " " + question else: prompt = DEFAULTIMAGETOKEN + " " + question

conv = convtemplates["qwen15"].copy() # Use the appropriate conversation template conv.appendmessage(conv.roles[0], prompt) conv.appendmessage(conv.roles[1], None) promptfinal = conv.get_prompt()

Load and process video frames

print("Loading video...") videoframes = loadvideo(videolocalpath, maxframesnum=maxframes) print(f"Video loaded, shape: {videoframes.shape}")

Preprocess video frames

print("Preprocessing video...")

Ensure video has shape (T, H, W, C) before preprocessing

videotensor = imageprocessor.preprocess(videoframes, returntensors="pt")["pixelvalues"] videotensor = videotensor.to(model.device, dtype=torch.bfloat16) videos = [videotensor] # The model expects a list of video tensors print(f"Video tensor processed, shape: {videos[0].shape}")

Tokenize the prompt

inputids = tokenizerimagetoken(promptfinal, tokenizer, IMAGETOKENINDEX, returntensors='pt') inputids = inputids.to(device=model.device, nonblocking=True)

Add batch dimension if necessary (tokenizerimagetoken might already return batched)

if inputids.ndim == 1: inputids = inputids.unsqueeze(0) print(f"Input IDs processed, shape: {inputids.shape}")

Generate response

print("Generating response...") with torch.inferencemode(): outputids = model.generate( inputids, images=videos, # Pass the processed video tensor list dosample=True, temperature=0.2, topp=1.0, numbeams=1, maxnewtokens=1024, use_cache=True )

Decode and print the output

outputs = tokenizer.batchdecode(outputids, skipspecialtokens=True)[0].strip() print(f" User input: {question} ") print(f"Model output: {outputs}")

## License

The model weights are released under the [CC-BY-NC-4.0 license](LICENSE).
The code is released under the Apache 2.0 license.
Users must comply with all terms and conditions of the original licenses, including the specific licenses for the base language model ([Qwen2 License](https://huggingface.co/Qwen/Qwen2-7B-Instruct/blob/main/LICENSE)).

## Citation

If you find this work useful, please consider citing the paper:

@misc{zhou2025slowfast, title={Slow-Fast Architecture for Video Multi-Modal Large Language Models}, author={Yifei Zhou and Jiaming Zuo and Chen Change Loy and Chongyang Zhong and Xin Wang and Qi Wu and Weidong Cai and Xiaodong He and Qingzhong Wang and Lei Zhang and Marcelo H. Ang Jr and Boyang Li and Yanfeng Wang and Qinghai He and Fengbei Liu and Liangchen Luo and Jingdong Wang and Conghui He and Wenhai Wang}, year={2025}, eprint={2504.01328}, archivePrefix={arXiv}, primaryClass={cs.CV} }

*(Note: Author list based on potential updates to the arXiv paper; please verify with the final published version if available.)*