theFoxofSky/RealisDance-DiT

RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wild

<a href='https://arxiv.org/abs/2504.14977'> <img src='https://img.shields.io/badge/arXiv-red'></a> <a href='https://thefoxofsky.github.io/project_pages/RealisDance-DiT/index'> <img src='https://img.shields.io/badge/Project-Page-orange'></a> <a href='https://huggingface.co/datasets/theFoxofSky/RealisDance-Val'> <img src='https://img.shields.io/badge/Test-Dataset-blue'></a> <a href='https://github.com/damo-cv/RealisDance'> <img src='https://img.shields.io/badge/Git-Hub-green'></a>

This repository is the official model checkpoint of RealisDance-DiT. RealisDance-DiT is a structurally simple, empirically robust, and experimentally strong baseline model for controllable character animation in the wild.

Gallery

Here are several character animation generated by RealisDance-DiT. Note that the GIFs shown here have some degree of visual quality degradation. Please visit our project page for more original videos

<table class="center"> <tr> <td><img src="https://github.com/damo-cv/RealisDance/blob/main/_assets/samples/1.gif?raw=true"></td> <td><img src="https://github.com/damo-cv/RealisDance/blob/main/assets/samples/2.gif?raw=true"></td> </tr> <tr> <td><img src="https://github.com/damo-cv/RealisDance/blob/main/assets/samples/3.gif?raw=true"></td> <td><img src="https://github.com/damo-cv/RealisDance/blob/main/assets/samples/4.gif?raw=true"></td> </tr> <tr> <td><img src="https://github.com/damo-cv/RealisDance/blob/main/assets/samples/5.gif?raw=true"></td> <td><img src="https://github.com/damo-cv/RealisDance/blob/main/assets_/samples/6.gif?raw=true"></td> </tr> </table>

Quick Start

1. Setup Repository and Environment

git clone https://github.com/theFoxofSky/RealisDance.git
cd RealisDance

conda create -n realisdance python=3.10
conda activate realisdance

pip install -r requirements.txt

# FA3 (Optional)
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git checkout ea3ecea97a1393c092863330aff9a162bb5ce443  # very important, using other FA3 will yield bad results
cd hopper
python setup.py install

2. Quick Inference

• —Inference with Demo sequences

python inference.py \
    --ref __assets__/demo/ref.png \
    --smpl __assets__/demo/smpl.mp4 \
    --hamer __assets__/demo/hamer.mp4 \
    --prompt "A blonde girl is doing somersaults on the grass. Behind the grass is a river, \
    and behind the river are trees and mountains. The girl is wearing black yoga pants and a black sports vest." \
    --save-dir ./output

Disclaimer

This project is released for academic use. We disclaim responsibility for user-generated content.

Contact Us

Jingkai Zhou: fs.jingkaizhou@gmail.com

BibTeX

@article{zhou2025realisdance-dit,
  title={RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wild},
  author={Zhou, Jingkai and Wu, Yifan and Li, Shikai and Wei, Min and Fan, Chao and Chen, Weihua and Jiang, Wei and Wang, Fan},
  journal={arXiv preprint arXiv:2504.14977},
  year={2025}
}

@article{zhou2024realisdance,
  title={RealisDance: Equip controllable character animation with realistic hands},
  author={Zhou, Jingkai and Wang, Benzhi and Chen, Weihua and Bai, Jingqi and Li, Dongyang and Zhang, Aixi and Xu, Hao and Yang, Mingyang and Wang, Fan},
  journal={arXiv preprint arXiv:2409.06202},
  year={2024}
}

Acknowledgements

Thanks Shikai Li for condition paraperation and Chenjie Cao for pose align.