Apppublic

aleafy/RelightVid

sourceHugging Facemitupdated 1y agoView on Hugging Face

2likes

App README

RelightVid

[RelightVid: Temporal-Consistent Diffusion Model for Video Relighting](https://arxiv.org/abs/2501.16330) Ye Fang\, [Zeyi Sun](https://github.com/SunzeY)\, Shangzhan Zhang, Tong Wu, Yinghao Xu, Pan Zhang, Jiaqi Wang, Gordon Wetzstein, Dahua Lin

*Equal Contribution <a href="https://arxiv.org/abs/2501.16330"><img src="https://img.shields.io/badge/arXiv-Paper-<color>"></a> <a href="https://sunzey.github.io/Make-it-Real"><img src="https://img.shields.io/badge/Project-Website-red"></a> <a href="https://www.youtube.com/watch?v=j-t8592GCM"><img src="https://img.shields.io/static/v1?label=Demo&message=Video&color=orange"></a> <a href="" target='blank'> <img src="https://visitor-badge.laobi.icu/badge?pageid=Aleafy.RelightVid&leftcolor=gray&right_color=blue"> </a>

[image]

📜 News

<!-- 🚀 [2024/6/8] We release our inference pipeline of Make-it-Real, including material matching and generation of albedo-only 3D objects.

🚀 [2024/6/8] Material library annotations generated by GPT-4V and data engine are released! -->

✨ [2025/3/12] The inference code, project page and huggingface demo are released!

✨ [2025/1/27] We release the paper of RelightVid!

💡 Highlights

—🔥 We first demonstrate that GPT-4V can effectively recognize and describe materials, allowing our model to precisely identifies and aligns materials with the corresponding components of 3D objects.
—🔥 We construct a Material Library containing thousands of materials with highly detailed descriptions readily for MLLMs to look up and assign.
—🔥 An effective pipeline for texture segmentation, material identification and matching, enabling the high-quality application of materials to 3D assets.

👨‍💻 Todo

—[ ] Evaluation for Existed and Model-Generated Assets (both code & test assets)
—[ ] More Interactive Demos (huggingface, jupyter)
—[x] Make-it-Real Pipeline Inference Code
—[x] Highly detailed Material Library annotations (generated by GPT-4V)
—[x] Paper and Web Demos

💾 Installation

bash

   git clone https://github.com/Aleafy/RelightVid.git
   cd RelightVid

   conda create -n relitv python=3.10 
   conda activate relitv
   
   pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu118
   pip install -r requirements.txt

📦 Data Preparation

1.Annotations: in data/material_lib/annotations folder, include:
2.Highly-detailed descriptions by GPT-4V: offering thorough descriptions of the material’s visual characteristics and rich semantic information.
3.Category-tree: Divided into a hierarchical structure with coarse and fine granularity, it includes over 80 subcategories.
4.PBR Maps: You can download the complete PBR data collection at Huggingface, or download the data used in our project at OpenXLab (Recommended). (If you have any questions, please refer to issue#5)
5.Material Images(optinal): You can download the material images file here, to check and visualize the material appearance.

<pre> MakeitReal └── data └── materiallib ├── annotations ├── matimages └── pbr_maps └── train ├── Ceremic ├── Concrete ├── ... └── Wood </pre>

⚡ Quick Start

Inference

bash

python main.py --obj_dir <object_dir> --exp_name <unique_exp_name> --api_key <your_own_gpt4_api_key>

—To ensure proper network connectivity for GPT-4V, add proxy environment settings in main.py (optional). Also, please verify the reachability of your API host.
—Result visualization (blender engine) is located in the output/refine_output dir. You can compare the result with that in output/ori_output.

Annotation Engine

bash

cd scripts/gpt_anno
python gpt4_query_mat.py

Note: Besides functinoning as annotation engine, you can also use this code (gpt4_query_mat.py) to test the GPT-4V connection simply.

❤️ Acknowledgments

—MatSynth: a Physically Based Rendering (PBR) materials dataset, which offers extensive high-resolusion tilable pbr maps to look up.
—TEXTure: Wonderful text-guided texture generation model, and the codebase we built upon.
—SoM: Draw visual cues on images to facilate GPT-4V query better.
—Material Palette: Excellent exploration of material extraction and generation, offers good insights and comparable setting.

✒️ Citation

If you find our work helpful for your research, please consider giving a star ⭐ and citation 📝

bibtex

@misc{fang2024makeitreal,
      title={Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials}, 
      author={Ye Fang and Zeyi Sun and Tong Wu and Jiaqi Wang and Ziwei Liu and Gordon Wetzstein and Dahua Lin},
      year={2024},
      eprint={2404.16829},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}