precison9/avatar-animation-pipeline

🎭 Video-Driven Avatar Animation Pipeline

Production-quality system that captures real human facial performance from video and retargets it onto a stylized avatar.

Architecture

Input Video (MP4, front-facing, well-lit)
    │
    ▼
[1] Face Detection → OpenCV Haar Cascade (CPU, no GPU)
    │
    ▼
[2] 68 Landmark Extraction → eyes, brows, nose, mouth, jaw
    │
    ▼
[3] Head Pose + Expression Analysis
    │
    ▼
[4] Temporal Smoothing (windowed averaging)
    │
    ▼
[5] Piecewise Affine Warping → Delaunay triangular mesh deformation
    │
    ▼
[6] Video Render → MP4 at 1080p/720p

Key Features

• —Pure CPU — No GPU, no MediaPipe GL dependencies, works everywhere
• —68-point face model — Industry standard, compatible with Blender/UE
• —Delaunay mesh warping — Piecewise affine transforms for smooth deformation
• —Temporal smoothing — Eliminates jitter, preserves natural timing
• —Canonical landmark fallback — Works with cartoon/drawn avatars (no face detection needed)
• —Procedural avatar generation — Build characters with color pickers if no reference image

Usage

1. 1.Upload Input Video — Front-facing speaker, good lighting (the motion source)
2. 2.Upload Avatar Image OR customize procedural avatar with colors/hair/expression
3. 3.Click Preview Avatar to see the character
4. 4.Adjust Expression Exaggeration (1.2 = slightly more animated than source)
5. 5.Click Process Video — wait 1-2 min per minute of footage
6. 6.Download: animated video + motion data JSON + Blender import script

Pipeline Parameters

Limitations

• —Lip sync uses video mouth shapes, not audio phonemes. For audio-driven precision, add a dedicated module (e.g., Rhubarb, Wav2Lip).
• —No full body — face/head only. Add DWPose or MediaPipe Holistic for shoulders.
• —No audio on HF Spaces (ffmpeg not available). Run locally for audio.
• —Haar detection can miss faces in poor lighting or extreme angles. Use front-facing, well-lit footage.

Tech Stack

• —Python 3.10+, OpenCV 4.8+, NumPy, Gradio 6
• —No deep learning models, no GPU required
• —Optional: FFmpeg for audio extraction

Local Development

pip install -r requirements.txt
python app.py

Extending