CoolFace
Apppublic

Hazeezadebayo/doppelgen

sourceHugging Faceupdated 26d agoView on Hugging Face
0likes
App README

Talkinghead Orchestrator

Flora serves as the centralized master director and orchestration service for the Creatorium suite (Voxa, Sonora, Scenea, Actora, Captiona, Ocula, Tempora). Powered by Google ADK (`google-adk`) and Gemini 3.5 Flash, it unifies deep learning models, computer vision tools, audio processing, NLP sub-services, and social publishing into a single containerized cloud runtime.

Google Cloud Infrastructure & ADK Multi-Agent Architecture

mermaid
graph TD
    A[User / Single-Page Studio UI] -->|HTTPS POST| B[Google Cloud Run - Flora API Container]
    B -->|Master Orchestrator| C[Flora ADK Director Agent - Gemini 3.5 Flash]
  
    C -->|Persists Job State| D[(Google Cloud Firestore DB)]
    C -->|Stores Media Assets| E[(Google Cloud Storage GCS Bucket)]
    C -->|Streams Reasoning Spans| F[OpenTelemetry Telemetry Dashboard]
  
    C -->|1. Voxa TTS| G[Speech Waveform & Transcript]
    C -->|2. Sonora Music| H[Ambient Soundtracks]
    C -->|3. Scenea B-roll| I[Context B-rolls]
    C -->|4. Actora LipSync| J[LipSync 1080p Video]
    C -->|5. Captiona Subtitles| K[Dynamic Styled Subtitles]
    C -->|6. Tempora Publisher| L[YouTube / Twitter / Playwright CDP Social Media]

Quick Start — Deploy to Google Cloud Run

Flora is pre-configured for 1-click cloud deployment on Google Cloud Run using Firestore and Google Cloud Storage (GCS):

bash
# Set GCP Project Environment Variables
export GCP_PROJECT_ID="your-gcp-project-id"
export GEMINI_API_KEY="your-gemini-api-key"

# Deploy to Google Cloud Run via master runner
./run_flora.sh deploy

High-Level Architectural Flow

mermaid
graph TD
    A[Frontend UI] -->|POST Form Data / File paths| B[Flora API]
    B -->|1. Voxa Runner| C[TTS Waveform & Transcript]
    B -->|2. Sonora Runner| D[Ambient Soundtracks]
    B -->|3. Scenea Runner| E[Context B-rolls]
    B -->|4. Actora Runner| F[LipSync Talking Head]
    B -->|5. Captiona Runner| G[Composed Subtitled Video]
    G -->|Success Payload| A
    B -.->|GET Status Poll| A

The Creatorium Pipeline

Flora features a unified Web UI (flora/flora/web) that serves as a single-page pipeline, sequentially triggering the independent nodes below:

  1. 1.Sonora
  • —Input: [speech.txt] + [optional speech.wav]
  • —Process: Performs semantic similarity analysis on the speech text (and speech.wav if provided) to identify the ideal background context, then anlyze pulled high-quality, royalty-free audio tracks from Mixkit, Freesound, and OpenGameArt for alignment.
  • —Output: Background ambient audio track background.wav.
  • —Voxa
  • —Input: [speech.txt] + [sample_audio.wav]
  • —Process: Utilizes advanced ASR and TTS models to clone the provided voice and narrate the speech text.
  • —Output: timestamped_transcript.txt and speech.wav (the cloned narration).
  • —Scenea
  • —Input: [timestamped_transcript.txt]
  • —Process: Analyzes the transcript to determine which segments require visual enhancement. It fetches relevant, concise B-roll videos (from Pexels or custom sets) tailored perfectly to those specific speech segments (e.g., generating 2 B-rolls if the user specifies a limit of 2).
  • —Output: B-roll video assets tightly bound to their transcript timestamps.
  • —Actora
  • —Input: [background.jpg] + [me.jpg] + [driving_video.mp4] + [speech.wav]
  • —Process: Fuses the assets together, applying human-like mannerisms extracted from the driving video to the static image of "me". The lip-syncing is perfectly matched to speech.wav.
  • —Output: talkinghead.mp4 (A complete, high-fidelity talking head video).
  • —Captiona
  • —Input: [talkinghead.mp4] + [timestamped_transcript.txt]
  • —Process: Overlays dynamic, styled text captions onto the video, perfectly synchronized with the speech and configured to the user's stylistic preferences.
  • —Output: talkinghead_captioned.mp4 (The final, ready-to-publish video).

Pipeline Parallelization Plan

We will optimize the execution speed of the Flora orchestration pipeline by running independent tasks concurrently.

Parallel Execution Architecture

Currently, the pipeline runs sequentially:

Voxa (TTS) -> Sonora (Ambient Audio) -> Scenea (B-rolls) -> Actora (Talking Head) -> Captiona (Subtitles)

However, after Voxa runs and produces the speech.wav and speech_transcript.txt files, the subsequent stages have no data dependencies on each other:

  • —Sonora only depends on speech_transcript.txt.
  • —Scenea only depends on speech_transcript.txt.
  • —Actora only depends on speech.wav.

Thus, we can execute Sonora, Scenea, and Actora concurrently using Python's concurrent.futures.ThreadPoolExecutor.

mermaid
graph TD
    A[Voxa TTS] --> B[Sonora Ambient Audio]
    A --> C[Scenea B-rolls]
    A --> D[Actora Talking Head]
    B --> E[Captiona Subtitles]
    C --> E
    D --> E

Impact on Execution Time

The total execution time will drop from: Time(Voxa) + Time(Sonora) + Time(Scenea) + Time(Actora) + Time(Captiona) to: Time(Voxa) + max(Time(Sonora), Time(Scenea), Time(Actora)) + Time(Captiona)

With the host's 32-core CPU capacity, running these three processes simultaneously will not bottleneck local resources, leading to a substantial performance improvement.

Review

[!IMPORTANT] Because subprocesses are run concurrently, stdout and stderr logs will write to the container console in an interleaved manner. However, each sub-process will still run as an isolated execution thread and write to its own independent logs if needed. We will update PIPELINE_STATUS to show active progress for all running components (e.g. "Generating Video & Fetching Assets...").

Similar apps:

Flora expects a strict Input/Output contract to seamlessly pass data between the nodes:

  • —Talking head gen:
  • —https://www.veed.io/tools/text-to-speech-avatar/talking-head-video
  • —https://www.synthesia.io/tools/talking-head-video-maker
  • —https://toki.ai/ai-talking-avatar
  • —/https://captions.ai/solutions/talking-head-videos
  • —Video understanding:
  • —/https://huggingface.co/openai/clip-vit-base-patch32/tree/main
  • —/https://huggingface.co/google/siglip-base-patch16-224/tree/main
  • —/https://huggingface.co/microsoft/xclip-base-patch32/tree/main
  • —/https://huggingface.co/apple/MobileCLIP2-S3/tree/main

By treating flora as the orchestrator, the entire Creatorium ecosystem operates as a cohesive, highly-optimized production engine.