pengsida/NeuralBody
1
1**News**2 3* `05/17/2021` To make the comparison on ZJU-MoCap easier, we save quantitative and qualitative results of other methods at [here](https://github.com/zju3dv/neuralbody/blob/master/supplementary_material.md#results-of-other-methods-on-zju-mocap), including Neural Volumes, Multi-view Neural Human Rendering, and Deferred Neural Human Rendering.4* `05/13/2021` To make the following works easier compare with our model, we save our rendering results of ZJU-MoCap at [here](https://zjueducn-my.sharepoint.com/:u:/g/personal/pengsida_zju_edu_cn/Ea3VOUy204VAiVJ-V-OGd9YBxdhbtfpS-U6icD_rDq0mUQ?e=cAcylK) and write a [document](supplementary_material.md) that describes the training and test protocols.5* `05/12/2021` The code supports the test and visualization on unseen human poses.6* `05/12/2021` We update the ZJU-MoCap dataset with better fitted SMPL using [EasyMocap](https://github.com/zju3dv/EasyMocap). We also release a [website](https://zju3dv.github.io/zju_mocap/) for visualization. Please see [here](https://github.com/zju3dv/neuralbody#potential-problems-of-provided-smpl-parameters) for the usage of provided smpl parameters.7 8# Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans9### [Project Page](https://zju3dv.github.io/neuralbody) | [Video](https://www.youtube.com/watch?v=BPCAMeBCE-8) | [Paper](https://arxiv.org/pdf/2012.15838.pdf) | [Data](https://github.com/zju3dv/neuralbody/blob/master/INSTALL.md#zju-mocap-dataset)10 1112 13> [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://arxiv.org/pdf/2012.15838.pdf) 14> Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou 15> CVPR 202116 17Any questions or discussions are welcomed!18 19## Installation20 21Please see [INSTALL.md](INSTALL.md) for manual installation.22 23### Installation using docker24 25Please see [docker/README.md](docker/README.md).26 27Thanks to [Zhaoyi Wan](https://github.com/wanzysky) for providing the docker implementation.28 29## Run the code on the custom dataset30 31Please see [CUSTOM](tools/custom).32 33## Run the code on People-Snapshot34 35Please see [INSTALL.md](INSTALL.md) to download the dataset.36 37We provide the pretrained models at [here](https://zjueducn-my.sharepoint.com/:f:/g/personal/pengsida_zju_edu_cn/Enn43YWDHwBEg-XBqnetFYcBLr3cItZ0qUFU-oKUpDHKXw?e=FObjE9).38 39### Process People-Snapshot40 41We already provide some processed data. If you want to process more videos of People-Snapshot, you could use [tools/process_snapshot.py](tools/process_snapshot.py).42 43You can also visualize smpl parameters of People-Snapshot with [tools/vis_snapshot.py](tools/vis_snapshot.py).44 45### Visualization on People-Snapshot46 47Take the visualization on `female-3-casual` as an example. The command lines for visualization are recorded in [visualize.sh](visualize.sh).48 491. Download the corresponding pretrained model and put it to `$ROOT/data/trained_model/if_nerf/female3c/latest.pth`.502. Visualization:51 * Visualize novel views of single frame52 ```53 python run.py --type visualize --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c vis_novel_view True num_render_views 14454 ```55 56 57 58 * Visualize views of dynamic humans with fixed camera59 ```60 python run.py --type visualize --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c vis_novel_pose True61 ```62 63 64 65 * Visualize mesh66 ```67 # generate meshes68 python run.py --type visualize --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c vis_mesh True train.num_workers 069 # visualize a specific mesh70 python tools/render_mesh.py --exp_name female3c --dataset people_snapshot --mesh_ind 22671 ```72 73 74 753. The results of visualization are located at `$ROOT/data/render/female3c` and `$ROOT/data/perform/female3c`.76 77### Training on People-Snapshot78 79Take the training on `female-3-casual` as an example. The command lines for training are recorded in [train.sh](train.sh).80 811. Train:82 ```83 # training84 python train_net.py --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c resume False85 # distributed training86 python -m torch.distributed.launch --nproc_per_node=4 train_net.py --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c resume False gpus "0, 1, 2, 3" distributed True87 ```882. Train with white background:89 ```90 # training91 python train_net.py --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c resume False white_bkgd True92 ```933. Tensorboard:94 ```95 tensorboard --logdir data/record/if_nerf96 ```97 98## Run the code on ZJU-MoCap99 100Please see [INSTALL.md](INSTALL.md) to download the dataset.101 102We provide the pretrained models at [here](https://zjueducn-my.sharepoint.com/:f:/g/personal/pengsida_zju_edu_cn/Enn43YWDHwBEg-XBqnetFYcBLr3cItZ0qUFU-oKUpDHKXw?e=FObjE9).103 104### Potential problems of provided smpl parameters105 1061. The newly fitted parameters locate in `new_params`. Currently, the released pretrained models are trained on previously fitted parameters, which locate in `params`.1072. The smpl parameters of ZJU-MoCap have different definition from the one of MPI's smplx.108 * If you want to extract vertices from the provided smpl parameters, please use `zju_smpl/extract_vertices.py`.109 * The reason that we use the current definition is described at [here](https://github.com/zju3dv/EasyMocap/blob/master/doc/02_output.md#attention-for-smplsmpl-x-users).110 111It is okay to train Neural Body with smpl parameters fitted by smplx.112 113### Test on ZJU-MoCap114 115The command lines for test are recorded in [test.sh](test.sh).116 117Take the test on `sequence 313` as an example.118 1191. Download the corresponding pretrained model and put it to `$ROOT/data/trained_model/if_nerf/xyzc_313/latest.pth`.1202. Test on training human poses:121 ```122 python run.py --type evaluate --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313123 ```1243. Test on unseen human poses:125 ```126 python run.py --type evaluate --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 test_novel_pose True127 ```128 129### Visualization on ZJU-MoCap130 131Take the visualization on `sequence 313` as an example. The command lines for visualization are recorded in [visualize.sh](visualize.sh).132 1331. Download the corresponding pretrained model and put it to `$ROOT/data/trained_model/if_nerf/xyzc_313/latest.pth`.1342. Visualization:135 * Visualize novel views of single frame136 ```137 python run.py --type visualize --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 vis_novel_view True138 ```139 140 141 * Visualize novel views of single frame by rotating the SMPL model142 ```143 python run.py --type visualize --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 vis_novel_view True num_render_views 100144 ```145 146 147 * Visualize views of dynamic humans with fixed camera148 ```149 python run.py --type visualize --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 vis_novel_pose True num_render_frame 1000 num_render_views 1150 ```151  152 153 * Visualize views of dynamic humans with rotated camera154 ```155 python run.py --type visualize --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 vis_novel_pose True num_render_frame 1000156 ```157 158 159 * Visualize mesh160 ```161 # generate meshes162 python run.py --type visualize --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 vis_mesh True train.num_workers 0163 # visualize a specific mesh164 python tools/render_mesh.py --exp_name xyzc_313 --dataset zju_mocap --mesh_ind 0165 ```166 167 1684. The results of visualization are located at `$ROOT/data/render/xyzc_313` and `$ROOT/data/perform/xyzc_313`.169 170### Training on ZJU-MoCap171 172Take the training on `sequence 313` as an example. The command lines for training are recorded in [train.sh](train.sh).173 1741. Train:175 ```176 # training177 python train_net.py --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 resume False178 # distributed training179 python -m torch.distributed.launch --nproc_per_node=4 train_net.py --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 resume False gpus "0, 1, 2, 3" distributed True180 ```1812. Train with white background:182 ```183 # training184 python train_net.py --cfg_file configs/zju_mocap_exp/latent_xyzc_313.yaml exp_name xyzc_313 resume False white_bkgd True185 ```1863. Tensorboard:187 ```188 tensorboard --logdir data/record/if_nerf189 ```190 191## Citation192 193If you find this code useful for your research, please use the following BibTeX entry.194 195```196@inproceedings{peng2021neural,197 title={Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans},198 author={Peng, Sida and Zhang, Yuanqing and Xu, Yinghao and Wang, Qianqian and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},199 booktitle={CVPR},200 year={2021}201}202```203