nvidia/PhysicalAI-SpatialIntelligence-Lyra-SDG

1---2license: cc-by-4.03task_categories:4- image-to-3d5- text-to-3d6tags:7- 3d-reconstruction8- gaussian-splatting9- video-diffusion10- synthetic-data11---12 13# Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation14 15**[Paper](https://arxiv.org/abs/2509.19296), [Project Page](https://research.nvidia.com/labs/toronto-ai/lyra/), [Code](https://github.com/nv-tlabs/lyra)**16 17[Sherwin Bahmani](https://sherwinbahmani.github.io/),18[Tianchang Shen](https://www.cs.toronto.edu/~shenti11/),19[Jiawei Ren](https://jiawei-ren.github.io/),20[Jiahui Huang](https://huangjh-pub.github.io/),21[Yifeng Jiang](https://cs.stanford.edu/~yifengj/),22[Haithem Turki](https://haithemturki.com/),23[Andrea Tagliasacchi](https://theialab.ca/),24[David B. Lindell](https://davidlindell.com/),25[Zan Gojcic](https://zgojcic.github.io/),26[Sanja Fidler](https://www.cs.utoronto.ca/~fidler/),27[Huan Ling](https://www.cs.utoronto.ca/~linghuan/),28[Jun Gao](https://www.cs.utoronto.ca/~jungao/),29[Xuanchi Ren](https://xuanchiren.com/) <br>30 31## Dataset Description:32 33The PhysicalAI-SpatialIntelligence-Lyra-SDG Dataset is a multi-view 3D and 4D dataset generated using [GEN3C](https://github.com/nv-tlabs/GEN3C). 34The 3D reconstruction setup uses 59,031 images, while the 4D setup has 7,378 videos. All the data are from diverse text prompts, spanning various scenarios such as indoor and outdoor environments, humans, animals, and both realistic and imaginative content. We synthesize 6 camera trajectories for each image (3D) or video (4D), yielding 354,186 videos for the 3D and 44,268 videos for the 4D.35It contains videos in RGB and camera poses and depth of the videos.36 37This dataset is ready for commercial use.38 39## Dataset Owner(s):40NVIDIA Corporation41 42## Dataset Creation Date:432025/09/2344 45## License/Terms of Use: 46This dataset is licensed under the [Creative Commons Attribution 4.0 International License (CC-BY-4.0)](https://creativecommons.org/licenses/by/4.0/).47 48## Intended Usage:49Researchers and academics working in spatial intelligence problems can use it to train AI models for multi-view video generation or reconstruction.50 51## Dataset Characterization:52** Data Collection Method<br>53 [Synthetic] 54 55** Labeling Method<br>56[Synthetic]57 58## Dataset Format:59RGB in mp4, Camera pose in .npz, Depth in zip format60 61## Dataset Quantification:62The 3D reconstruction setup has 59,031 multi-view examples, while the 4D setup has 7,378 multi-view examples. For each multi-view example, we have 6 views. 63For each view, we have videos in Red, Green, Blue (RGB) and camera poses and depth of the videos.64 65 66| Field       | Format |67|-------------|--------|68| Video       | mp4    |69| Camera pose | .npz   |70| Depth       | .zip   |71 72Storage: 25TB73 74## Sample Usage75 76Lyra supports both images and videos as input for 3D Gaussian generation. First, you need to download the demo samples:77 78```bash79# Download test samples from Hugging Face80huggingface-cli download nvidia/Lyra-Testing-Example --repo-type dataset --local-dir assets/demo81```82 83### Example 1: Single Image to 3D Gaussians Generation84 851) Generate multi-view video latents from the input image using scripts/bash/static_sdg.sh. 86 87```bash88CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) torchrun --nproc_per_node=1 cosmos_predict1/diffusion/inference/gen3c_single_image_sdg.py \89    --checkpoint_dir checkpoints \90    --num_gpus 1 \91    --input_image_path assets/demo/static/diffusion_input/images/00172.png \92    --video_save_folder assets/demo/static/diffusion_output_generated \93    --foreground_masking \94    --multi_trajectory95```96 972) Reconstruct multi-view video latents with the 3DGS decoder:98 99```bash100accelerate launch sample.py --config configs/demo/lyra_static.yaml101```102 103### Example 2: Single Video to Dynamic 3D Gaussians Generation104 1051) Generate multi-view video latents from the input video and ViPE estimated depth using scripts/bash/dynamic_sdg.sh.106 107```bash108CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) torchrun --nproc_per_node=1 cosmos_predict1/diffusion/inference/gen3c_dynamic_sdg.py \109    --checkpoint_dir checkpoints \110    --vipe_path assets/demo/dynamic/diffusion_input/rgb/6a71ee0422ff4222884f1b2a3cba6820.mp4 \111    --video_save_folder assets/demo/dynamic/diffusion_output \112    --disable_prompt_upsampler \113    --num_gpus 1 \114    --foreground_masking \115    --multi_trajectory116```117 1182) Reconstruct multi-view video latents with the 3DGS decoder:119 120```bash121accelerate launch sample.py --config configs/demo/lyra_dynamic.yaml122```123 124### Training125 126To train, you need to download the full training data (this dataset) from Hugging Face:127 128```bash129# Download our training datasets from Hugging Face and untar them into a static/dynamic folder130huggingface-cli download nvidia/PhysicalAI-SpatialIntelligence-Lyra-SDG --repo-type dataset --local-dir lyra_dataset/tar131```132 133Then you can use the provided progressive training script (as detailed in the GitHub repository):134 135```bash136bash train.sh137```138 139For more detailed usage instructions, including how to test on your own videos or perform training, please refer to the [Lyra GitHub repository](https://github.com/nv-tlabs/lyra).140 141## Reference(s):142 143- [GEN3C](https://github.com/nv-tlabs/GEN3C)144 145## Ethical Considerations:146NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.   147 148Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).149 150## Citation151```152@inproceedings{bahmani2025lyra,153        title={Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation},154        author={Bahmani, Sherwin and Shen, Tianchang and Ren, Jiawei and Huang, Jiahui and Jiang, Yifeng and 155            Turki, Haithem and Tagliasacchi, Andrea and Lindell, David B. and Gojcic, Zan and Fidler, Sanja and 156            Ling, Huan and Gao, Jun and Ren, Xuanchi},157        booktitle={arXiv preprint arXiv:2509.19296},158        year={2025}159}160```161 162```163@inproceedings{ren2025gen3c,164    title={GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control},165    author={Ren, Xuanchi and Shen, Tianchang and Huang, Jiahui and Ling, Huan and166        Lu, Yifan and Nimier-David, Merlin and M\u00fcller, Thomas and Keller, Alexander and167        Fidler, Sanja and Gao, Jun},168    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},169    year={2025}170}171```