hugging-apps/4danyone-multiview-demo

4DAnyone — one video in, a camera rig out

Interactive demo of **AntResearch/4DAnyone**: a single monocular portrait video goes in, a set of synchronized multi-view videos of the same performance comes out — the input a 4D Gaussian-splatting reconstruction needs.

Pipeline

1. 1.Canonical clip — 121 frames are decoded at a fixed CFR clock.
2. 2.Motion — GVHMR (YOLOv8x tracker → ViTPose-H → HMR2.0a features → GVHMR transformer) recovers world-grounded SMPL-X motion from the static-camera clip.
3. 3.Conditioning — BiRefNet mattes the subject, an MHR/Goliath-70 regressor turns SMPL-X into a dense skeleton, and the skeleton is rendered from a virtual camera ring solved to match the input framing.
4. 4.Generation — a Wan2.2-TI2V-5B-derived spatio-temporal DiT denoises all requested views jointly (multi-view self-attention + temporal-consistent routing), conditioned on the source video latents and the skeleton renders.
5. 5.Export — H.264 videos plus an OpenCV camera rig (cameras.json) and run metadata, zipped for download.

Fidelity

The demo runs the authors' frozen inference contract unchanged: 121 frames, 1280×704, 24 steps, scheduler shift 5.0, denoising strength 1.0, untiled VAE, bf16 autocast, and the fixed prompt 视频中的人在做动作.

Two deployment-only deviations:

• —the reference CLI runs GVHMR and skeleton rendering in subprocesses; a ZeroGPU worker cannot fork a second CUDA process, so they run in-process;
• —the 11.4 GB UMT5-XXL text encoder is replaced by the pre-computed embedding of that single fixed prompt (prompt_embedding.safetensors), which is what the encoder would have produced. The DiT, VAE and everything else are the released weights, resident on the GPU (no CPU offloading).

Notes

• —Inputs should be roughly 9:16 portrait video of a single person, at least 121 frames long from the chosen start time.
• —Up to six views are generated directly; the released RCP proposal stage only engages above six views and is therefore inert here.
• —A four-view run takes about 9 minutes of ZeroGPU time and peaks at 53 GiB of VRAM, so it runs on the 96 GB ZeroGPU tier and needs a PRO-tier (or higher) quota. Fewer views are proportionally faster.

Credits & licences

• —Model + code: ant-research/4DAnyone, Apache-2.0 (vendored as fdanyone/, see LICENSE).
• —Motion recovery: GVHMR — cloned at runtime, research use only.
• —Foreground matting: ZhengPeng7/BiRefNet, MIT.
• —Body model: SMPL-X (Max Planck Institute) — research licence; see <https://smpl-x.is.tue.mpg.de/>.
• —Object detection weights: YOLOv8x (Ultralytics), AGPL-3.0.
• —Example clips: Pexels, as redistributed in the 4DAnyone model repository.