brjathu/HMR2.0

1# 4DHumans: Reconstructing and Tracking Humans with Transformers2Code repository for the paper:3**Humans in 4D: Reconstructing and Tracking Humans with Transformers**4[Shubham Goel](https://people.eecs.berkeley.edu/~shubham-goel/), [Georgios Pavlakos](https://geopavlakos.github.io/), [Jathushan Rajasegaran](http://people.eecs.berkeley.edu/~jathushan/), [Angjoo Kanazawa](https://people.eecs.berkeley.edu/~kanazawa/)<sup>\*</sup>, [Jitendra Malik](http://people.eecs.berkeley.edu/~malik/)<sup>\*</sup>5arXiv preprint 20236[[paper]()] [[project page](https://shubham-goel.github.io/4dhumans/)] [[hugging faces space]()]7 8![teaser](assets/teaser.png)9 10## Download dependencies11Our demo code depends on [detectron2](https://github.com/facebookresearch/detectron2) to detect humans.12To automatically download this dependency, clone this repo using `--recursive`, or run `git submodule update --init` if you've already cloned the repository. You should see the detectron2 source code at `vendor/detectron2`.13```bash14git clone https://github.com/shubham-goel/4D-Humans.git --recursive15# OR16git clone https://github.com/shubham-goel/4D-Humans.git17cd 4D-Humans18git submodule update --init19```20 21## Installation22We recommend creating a clean [conda](https://docs.conda.io/) environment and installing all dependencies, as follows:23```bash24conda env create -f environment.yml25```26 27After the installation is complete you can activate the conda environment by running:28```29conda activate 4D-humans30```31 32## Download checkpoints and SMPL models33To download the checkpoints and SMPL models, run34```bash35./fetch_data.sh36```37 38## Run demo on images39You may now run our demo to 3D reconstruct humans in images using the following command, which will run ViTDet and HMR2.0 on all images in the specified `--img_folder` and save renderings of the reconstructions in `--out_folder`. You can also use the `--side_view` flag to additionally render the side view of the reconstructed mesh. `--batch_size` batches the images together for faster processing.40```bash41python demo.py \42    --img_folder example_data/images \43    --out_folder demo_out \44    --batch_size=48 --side_view45```46 47## Run demo on videos48Coming soon.49 50## Training and evaluation51Cmoing soon.52 53## Acknowledgements54Parts of the code are taken or adapted from the following repos:55- [ProHMR](https://github.com/nkolot/ProHMR)56- [SPIN](https://github.com/nkolot/SPIN)57- [SMPLify-X](https://github.com/vchoutas/smplify-x)58- [HMR](https://github.com/akanazawa/hmr)59- [ViTPose](https://github.com/ViTAE-Transformer/ViTPose)60- [Detectron2](https://github.com/facebookresearch/detectron2)61 62Additionally, we thank [StabilityAI](https://stability.ai/) for a generous compute grant that enabled this work.63 64## Citing65If you find this code useful for your research, please consider citing the following paper:66 67```68@article{4DHUMANS,69    title={Humans in 4{D}: Reconstructing and Tracking Humans with Transformers},70    author={Goel, Shubham and Pavlakos, Georgios and Rajasegaran, Jathushan and Kanazawa, Angjoo and Malik, Jitendra},71    journal={arXiv preprint},72    year={2023}73}74```75