LTT/PRM
24
1 2 3<div align="center">4 5# PRM: Photometric Stereo based Large Reconstruction Model6 7<a href="https://tau-yihouxiang.github.io/projects/X-Ray/X-Ray.html"><img src="https://img.shields.io/badge/Project_Page-Online-EA3A97"></a>8<a href="https://arxiv.org/abs/2404.07191"><img src="https://img.shields.io/badge/ArXiv-2404.07191-brightgreen"></a> 9<a href="https://huggingface.co/LTT/PRM"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Model_Card-Huggingface-orange"></a> <br>10<a href="https://huggingface.co/spaces/TencentARC/InstantMesh"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Gradio%20Demo-Huggingface-orange"></a>11<a href="https://github.com/jtydhr88/ComfyUI-InstantMesh"><img src="https://img.shields.io/badge/Demo-ComfyUI-8A2BE2"></a>12 13</div>14 15---16 17An official implementation of PRM, a feed-forward framework for high-quality 3D mesh generation with photometric stereo images.18 19 2021 22# 🚩 Features23- [x] Release inference and training code.24- [x] Release model weights.25- [x] Release huggingface gradio demo. Please try it at [demo](https://huggingface.co/spaces/TencentARC/InstantMesh) link.26- [x] Release ComfyUI demo.27 28# ⚙️ Dependencies and Installation29 30We recommend using `Python>=3.10`, `PyTorch>=2.1.0`, and `CUDA>=12.1`.31```bash32conda create --name PRM python=3.1033conda activate PRM34pip install -U pip35 36# Ensure Ninja is installed37conda install Ninja38 39# Install the correct version of CUDA40conda install cuda -c nvidia/label/cuda-12.1.041 42# Install PyTorch and xformers43# You may need to install another xformers version if you use a different PyTorch version44pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu12145pip install xformers==0.0.22.post746 47# Install Triton 48pip install triton49 50# Install other requirements51pip install -r requirements.txt52```53 54# 💫 Inference55 56## Download the pretrained model57 58The pretrained model can be found [model card](https://huggingface.co/LTT/PRM).59 60Our inference script will download the models automatically. Alternatively, you can manually download the models and put them under the `ckpts/` directory.61 62# 💻 Training63 64We provide our training code to facilitate future research. 65For training data, we used filtered Objaverse for training. Before training, you need to pre-processe the environment maps and GLB files into formats that fit our dataloader.66For preprocessing GLB files, please run67```bash68# GLB files to OBJ files69python train.py --base configs/instant-mesh-large-train.yaml --gpus 0,1,2,3,4,5,6,7 --num_nodes 170```71then 72```bash73# OBJ files to mesh files that can be readed74python obj2mesh.py path_to_obj save_path75```76For preprocessing environment maps, please run77```bash78# Pre-process environment maps79python light2map.py path_to_env save_path80```81 82 83To train the sparse-view reconstruction models, please run:84```bash85# Training on Mesh representation86python train.py --base configs/PRM.yaml --gpus 0,1,2,3,4,5,6,7 --num_nodes 187```88Note that you need to change to root_dir and light_dir to pathes that you save the preprocessed GLB files and environment maps.89 90# :books: Citation91 92If you find our work useful for your research or applications, please cite using this BibTeX:93 94```BibTeX95@article{xu2024instantmesh,96 title={InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models},97 author={Xu, Jiale and Cheng, Weihao and Gao, Yiming and Wang, Xintao and Gao, Shenghua and Shan, Ying},98 journal={arXiv preprint arXiv:2404.07191},99 year={2024}100}101```102 103# 🤗 Acknowledgements104 105We thank the authors of the following projects for their excellent contributions to 3D generative AI!106 107- [FlexiCubes](https://github.com/nv-tlabs/FlexiCubes)108- [InstantMesh]([https://instant-3d.github.io/](https://github.com/TencentARC/InstantMesh))109 110 111 112