OpenDriveLab/SparseVideoNav_VGM

1---2license: cc-by-nc-sa-4.03language: 4  - en5tags:6  - video-generation7  - vision-language-navigation8  - embodied-ai9  - pytorch10---11 12![SparseVideoNav Architecture](assets/caption.png)13# SparseVideoNav: Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation14 15## Model Details16 17### Model Description18 19SparseVideoNav introduces video generation models to real-world beyond-the-view vision-language navigation for the first time. It pioneers a paradigm shift from continuous to sparse video generation for longer prediction horizons. By guiding trajectory inference with a generated sparse future spanning a 20-second horizon, it achieves sub-second inference (a 27× speed-up). It also marks the first realization of beyond-the-view navigation in challenging night scenes.20 21- **Developed by:** Hai Zhang, Siqi Liang, Li Chen, Yuxian Li, Yukuan Xu, Yichao Zhong, Fu Zhang, Hongyang Li22- **Shared by:** The University of Hong Kong & OpenDriveLab23- **Model type:** Video Generation-based Model for Vision-Language Navigation24- **Language(s) (NLP):** English (Instruction prompts)25- **License:** CC BY-NC-SA 4.026- **Finetuned from model:** Based on UMT5-XXL (text encoder) and Wan2.1 VAE.27 28### Model Sources29 30- **Repository:** [https://github.com/OpenDriveLab/SparseVideoNav](https://github.com/OpenDriveLab/SparseVideoNav)31- **Paper:** [arXiv:2602.05827](https://arxiv.org/abs/2602.05827)32- **Project Page:** [https://opendrivelab.com/SparseVideoNav](https://opendrivelab.com/SparseVideoNav)33 34## Uses35 36### Direct Use37 38The model is designed for generating sparse future video frames based on a current visual observation (video) and a natural language instruction (e.g., "turn right"). It is primarily intended for research in Embodied AI, specifically Vision-Language Navigation (VLN) in real-world environments.39 40### Out-of-Scope Use41 42The model is a research prototype and is not intended for deployment in safety-critical real-world autonomous driving or robotic navigation systems without further extensive testing, safety validation, and fallback mechanisms.43 44## How to Get Started with the Model45 46Use the code below to get started with the model using our custom pipeline. 47 48Ensure you have cloned the [GitHub repository](https://github.com/OpenDriveLab/SparseVideoNav) and installed the requirements.49 50```python51from omegaconf import OmegaConf52from inference import SVNPipeline53 54# Load configuration55cfg = OmegaConf.load("config/inference.yaml")56cfg.ckpt_path = "/path/to/models/SparseVideoNav-Models" # Path to your downloaded weights57cfg.inference.device = "cuda:0"58 59# Initialize pipeline60pipeline = SVNPipeline.from_pretrained(cfg)61 62# Run inference (Returns np.ndarray (T, H, W, C) uint8)63video = pipeline(video="/path/to/input.mp4", text="turn right") 64```65 66## BibTeX67```python68@article{zhang2026sparse,69  title={Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation},70  author={Zhang, Hai and Liang, Siqi and Chen, Li and Li, Yuxian and Xu, Yukuan and Zhong, Yichao and Zhang, Fu and Li, Hongyang},71  journal={arXiv preprint arXiv:2602.05827},72  year={2026}73}