YuanzeLin/IllumiCraft

IllumiCraft Dataset

This repository contains the dataset released with:

IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation

Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Ronald Clark, Ming-Hsuan Yang

🔗 Links

• —📄 Paper: https://arxiv.org/abs/2506.03150
• —🌐 Project Page: https://yuanze-lin.me/IllumiCraft_page/
• —💻 GitHub: https://github.com/yuanze-lin/IllumiCraft
• —🎥 YouTube: https://youtu.be/qAV58sADEzo
• —🤗 Checkpoints: https://huggingface.co/YuanzeLin/Illumicraft-checkpoints

✨ Overview

The IllumiCraft dataset is designed for controllable video generation under varying geometry and illumination conditions. Each training sample contains multiple modalities used by IllumiCraft, including foreground appearance videos, background videos, tracking videos, lighting videos, and paired text descriptions.

🌟 Potential Applications

The IllumiCraft dataset provides paired geometry, appearance, and illumination control signals for video generation and can support a wide range of research directions, including:

• —🎥 Controllable video generation
• —💡 Video relighting and illumination editing
• —🎭 Appearance-preserving video synthesis
• —📍 Geometry-aware video generation
• —🎬 Video-to-video generation and editing
• —🌍 Virtual cinematography and lighting design
• —🤖 Multimodal video foundation models
• —🕶️ AR/VR content creation and scene manipulation

We hope the dataset will facilitate future research on controllable, physically plausible, and temporally consistent video generation.

📥 Download Dataset

Download the dataset using:

python utils/download_illumicraft_dataset.py

The dataset will be organized as:

dataset/
├── train/
└── demo_examples/

📂 Training Dataset

Example inputs for training are provided in:

dataset/train/
├── foreground_videos/
├── background_videos/
├── tracking_videos/
├── lighting_videos/
├── videos/
├── prompt.txt
├── videos.txt
├── foreground_videos.txt
├── background_videos.txt
├── tracking_videos.txt
└── lighting_videos.txt

Demo spotlight examples for inference are provided in:

dataset/demo_examples/
├── background_images/
├── foreground_videos/
├── background_images.txt
├── foreground_videos.txt
├── light.txt
└── prompt.txt

🚀 Usage

Please refer to the official GitHub repository for installation, dataset preparation, training, and inference instructions:

https://github.com/yuanze-lin/IllumiCraft

📖 Citation

If you find IllumiCraft useful for your research, please consider citing:

@article{lin2026illumicraft,
  title={IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation},
  author={Lin, Yuanze and Chen, Yi-Wen and Tsai, Yi-Hsuan and Clark, Ronald and Yang, Ming-Hsuan},
  journal={Advances in Neural Information Processing Systems},
  volume={38},
  pages={27798--27829},
  year={2026}
}