Modelpublic

piyushptiwari/bytical-2d-talk

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

1likes

Model Card

Bytical 2D Talk — Smart-AI Talking-Head System

Dimension: 2D · Renderer: SyncTalk_2D (mouth-inpaint lip-sync) This repo = the 2D _application_ (renderer + LLM "brain"). Public.

A talking-head video system that pairs a fast 2D lip-sync renderer with an LLM/embedding-driven brain that understands the script, adapts to any input video, and critiques its own output — so it performs a script instead of just reading it.

Where this sits in the Bytical family

Bytical has two independent talking-head stacks — a 2D one and a 3D one. This repo is the 2D application.

mermaid

graph TD
    ROOT["Bytical Talking-Head Systems"]
    ROOT --> TWOD["2D · SyncTalk_2D<br/>(mouth-inpaint lip-sync)"]
    ROOT --> THREED["3D · Gaussian Splatting<br/>(TalkingGaussian / InsTaG)"]

    TWOD --> A["bytical-2d-talk<br/>system + LLM brain · public"]
    TWOD --> B["bytical-2d-synctalk-archive<br/>R&D + trained weights · private"]
    THREED --> C["bytical-3d-head<br/>R&D checkpoints P1/P2 · private"]
    THREED --> D["bytical-3d-instag-pretrain<br/>multi-identity pretrain · public"]

    style A fill:#2563eb,color:#ffffff,stroke:#1e3a8a,stroke-width:3px

Repo	Dim	Role
bytical-2d-talk ← you are here	2D	System + LLM brain (this repo)
bytical-2d-synctalk-archive	2D	R&D lab notebook + trained weights/datasets
bytical-3d-head	3D	Gaussian-Splatting R&D checkpoints (P1/P2)
bytical-3d-instag-pretrain	3D	Multi-identity InsTaG pretrain weights

Why this exists

Vanilla lip-sync models take (video, audio) → video. That's a renderer, not intelligence. bytical-2d-talk adds a reasoning layer on top:

Brain module	Input → Output	What it means
Director	script → emotion/emphasis/pacing + SSML	reads meaning and decides delivery
AutoConfig	any video → render settings	no manual tuning; works on any clip
SelfQC	rendered video → pass/fail + fix	the system reviews itself and retries smarter

The renderer stays weight-safe and swappable; the brain is provider-agnostic (defaults to Azure OpenAI, works with any OpenAI-compatible endpoint).

End-to-end flow

mermaid

flowchart LR
    S["script"] --> DIR["Director<br/>(LLM)"]
    DIR --> PLAN["performance plan<br/>emotion timeline + SSML"]
    PLAN --> TTS["your TTS<br/>SSML → wav"]
    V["input video"] --> AC["AutoConfig<br/>(LLM/CV)"]
    AC --> CFG["render settings"]
    TTS --> R["render(video, wav, settings)<br/>SyncTalk_2D + improvements"]
    CFG --> R
    R --> MP4["mp4"]
    MP4 --> QC{"SelfQC<br/>pass?"}
    QC -- "no (bounded retry)" --> AC
    QC -- "yes" --> OUT["final mp4 + QC report"]

    style DIR fill:#8b5cf6,color:#fff
    style AC fill:#8b5cf6,color:#fff
    style QC fill:#f59e0b,color:#111
    style OUT fill:#16a34a,color:#fff

Reading the diagram: the brain nodes (purple) are LLM/CV steps that make decisions; the render step is the SyncTalk_2D renderer; SelfQC (amber) closes the loop by re-driving AutoConfig on failure, up to a bounded number of retries.

Package layout

Path	What it holds
`bytical_talk/brain/`	`llm.py`, `director.py`, `autoconfig.py`, `qc.py`
`bytical_talk/render/`	improved inference: One-Euro crop smoothing, feather paste-back, train/inference resize parity
`bytical_talk/audio/`	HuBERT features (better generalization to TTS voices)
`bytical_talk/losses/`	opt-in training upgrades (fixed VGG perceptual, mouth-weighted L1, PatchGAN, LPIPS)
`upstream/synctalk2d/`	the renderer, fetched by `scripts/fetch_upstream.sh` (not re-hosted)

Install

bash

git clone https://github.com/piyushptiwari1/bytical-talk.git
cd bytical-talk
pip install -e .            # brain only (light: openai, numpy, pyyaml)
pip install -e ".[render]"  # + renderer/audio/CV deps (torch, cv2, transformers, …)

cp .env.example .env        # then fill in your keys
bash scripts/fetch_upstream.sh   # only needed for rendering

Configure the brain

Edit .env (never committed). Default backend is Azure OpenAI:

BYTICAL_LLM_PROVIDER=azure
AZURE_OPENAI_ENDPOINT=https://<resource>.openai.azure.com/
AZURE_OPENAI_API_KEY=<key>
AZURE_DEPLOYMENT_NAME=gpt-4o-mini
AZURE_EMBEDDING_MODEL_NAME=text-embedding-3-small

Any OpenAI-compatible endpoint works with BYTICAL_LLM_PROVIDER=openai.

Use

bash

# verify credentials + upstream
bytical-talk env-check

# LLM performance plan (no GPU needed)
bytical-talk direct --script "We protect what matters most. Let's find your plan."

# analyze any video -> recommended render settings (needs [render])
bytical-talk autoconfig --video presenter.mp4

# quality review of a rendered clip
bytical-talk qc --video out.mp4

# full pipeline (needs a trained checkpoint + a wav)
bytical-talk generate --script "..." --checkpoint ckpt.pth \
  --dataset dataset/presenter --audio speech.wav --out out.mp4 --reference presenter.mp4

Python:

python

from bytical_talk import Director, auto_config, SelfQC

plan = Director().direct("Hi, I'm here to help you choose the right cover.")
print(plan.ssml)                 # Polly-ready SSML with emphasis + pauses
print(plan.emotion_timeline())   # per-sentence emotion for the renderer

Training a presenter (renderer)

Any short, front-facing talking clip works. Standard SyncTalk_2D flow, then infer with the improvements:

bash

python upstream/synctalk2d/data_utils/process.py dataset/<name>/<name>.mp4
python bytical_talk/audio/hubert.py --wav_path dataset/<name>/aud.wav --num_frames <N>   # for --asr hubert

The trained checkpoints, multi-identity base, and render-ready datasets live in the sibling archive bytical-2d-synctalk-archive (private).

Roadmap

Five pillars (see ROADMAP.md): quality (HuBERT ✓, FiLM multi-scale audio, attention fusion, temporal loss, super-res), speed (ONNX/TensorRT, fp16), expressiveness (emotion conditioning, gestures, prosody), self-learning (auto-QC ✓, hard-example mining, few-shot per-presenter adaptation), and optional, consent-gated swaps (background, face, voice, clothes — all default OFF).

Credits & license

—Renderer: ZiqiaoPeng/SyncTalk_2D (based on Ultralight-Digital-Human and SyncTalk) — fetched, not re-hosted.
—bytical_talk/ (the brain + improvements) is licensed Apache-2.0.
—Optional swap features must only be used on media you own or have rights to.