youngsong305/MV-S2V

MV-S2V: Multi-View Subject-Consistent Video Generation

<div align="center">

&nbsp; &nbsp; <a href="https://huggingface.co/youngsong305/MV-S2V"><img src="https://img.shields.io/static/v1?label=%F0%9F%A4%97%20Hugging%20Face&message=Model&color=orange"></a>&nbsp; <a href="https://huggingface.co/datasets/youngsong305/MV-S2V-Bench"><img src="https://img.shields.io/static/v1?label=%F0%9F%A4%97%20Hugging%20Face&message=Benchmark&color=yellow"></a>&nbsp; 

</div>

**MV-S2V: Multi-View Subject-Consistent Video Generation**<br> Ziyang Song¹, Xinyu Gong², Bangya Liu³, Zelin Zhao⁴ <br>¹The Hong Kong Polytechnic University &nbsp;&nbsp; ²The University of Texas at Austin &nbsp;&nbsp; ³University of Wisconsin–Madison &nbsp;&nbsp; ⁴Georgia Institute of Technology<br> <br>SIGGRAPH 2026<br>

<p align="center"> <img src="assets/overview.png" width="95%"> </p>

📖 Overview

MV-S2V is the first framework that synthesizes videos from multiple reference views of the same subject to enforce 3D-level subject consistency. Existing subject-to-video methods condition on a single reference image and are therefore forced to hallucinate unseen details when generating novel views. MV-S2V tackles this by:

• —🧊 Multi-view conditioning — accepts an arbitrary number of reference views and produces a coherent video that respects all of them.
• —🌀 Temporally-Shifted RoPE (TS-RoPE) — a positional-encoding scheme that disambiguates cross-subject and cross-view references.
• —📦 Synthetic + real-world dataset — a data curation pipeline for high-quality multi-view subject-to-video training data.

The model in this repo is a 14B Subject-to-Video DiT built on top of Wan2.1.

📑 Todo List

• —[x] Inference code
• —[x] Pretrained 14B checkpoint
• —[x] Benchmark dataset (MV-S2V-Bench)
• —[ ] Evaluation suite

⚡️ Quickstart

Installation

git clone https://github.com/szy-young/mv-s2v.git
cd mv-s2v
# Ensure torch >= 2.4.0
pip install -r requirements.txt
# For multi-GPU inference
pip install "xfuser>=0.4.1"

Model Download

MV-S2V re-uses the VAE and T5 text encoder from Wan2.1-T2V-14B. Download the base checkpoint first:

pip install "huggingface_hub[cli]"
huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./Wan2.1-T2V-14B

Then place our DiT weights at ./checkpoints/diffusion_pytorch_model[.safetensors|.safetensors.index.json].

Run Multi-View Subject-to-Video Generation

The simplest way to reproduce all examples in this repo:

bash infer.sh

infer.sh exposes two paths to edit:

WAN_CKPT="./Wan2.1-T2V-14B"               # Wan2.1 base (VAE + T5)
DIT_CKPT="./checkpoints/diffusion_pytorch_model"  # MV-S2V DiT weights

Single-prompt inference (multi-GPU FSDP + xDiT USP)

torchrun --nproc_per_node=2 --master-port 14435 generate.py \
    --task s2v-14B \
    --size 640*640 \
    --frame_num 121 --sample_fps 24 \
    --ckpt_dir ${WAN_CKPT} \
    --phantom_ckpt ${DIT_CKPT} \
    --dit_fsdp --t5_fsdp --ulysses_size 2 \
    --ref_image "examples/object_images/box_multi_door_colored/object_0.png,examples/object_images/box_multi_door_colored/object_1.png,examples/object_images/box_multi_door_colored/object_2.png,examples/object_images/box_multi_door_colored/object_3.png" \
    --prompt "The video starts with a small, wooden activity cube with colorful panels and metal locks, topped with a teal plastic handle, sitting on a child's playroom floor with a soft rug and scattered toys in the background. As the camera smoothly orbits around the activity cube, the background gradually reveals a low bookshelf filled with children's books and puzzles on one side, and a painted wall adorned with cheerful animal decals and a cozy reading nook with cushions on the other." \
    --save_file examples/videos/box_multi_door_colored.mp4 \
    --base_seed 42 \
    --rpe_mode ts_rope \
    --sample_guide_scale_img 2.5

Key arguments

<!-- ## 🖼️ Examples

The four configurations executed by infer.sh:

Generated videos are written to examples/videos/*.mp4. A larger gallery and qualitative comparisons against single-view baselines are on the project page. -->

🙏 Acknowledgements

This project builds on the open-source efforts of Wan2.1 and Phantom. We thank their authors for releasing high-quality video generation backbones.

⭐ Citation

If you find MV-S2V useful for your research, please cite our paper and ⭐ this repo.

@inproceedings{song2026mvs2v,
  title     = {MV-S2V: Multi-View Subject-Consistent Video Generation},
  author    = {Song, Ziyang and Gong, Xinyu and Liu, Bangya and Zhao, Zelin},
  booktitle = {ACM SIGGRAPH},
  year      = {2026}
}

📧 Contact

For questions or collaboration, please open a GitHub issue or contact Ziyang Song.