wrhuang/avatar-dialogue

Avatar Dialogue (v2)

A real-time voice chatbot avatar. Talk into your mic; pause for ~half a second; the avatar replies out loud and shows the reply text on its video frame. It's a fork of the v1 "sanity check" — same orchestrator ↔ model seam, but the stubs are now a real dialogue loop (streaming ASR → fast LLM → TTS + frame).

Pipeline

  user mic audio ──> DialogueDuplexModel ──> stable running transcript (words)
                                  │
                                  │  every 200ms tick:
                                  ▼
              ┌─────────────────────────────────────────────────────────┐
              │  drain newly-stable words into the pending user turn       │
              │  VAD: is the user still talking? (recent word activity)    │
              │  speculatively ask LLMResponder for a reply to the turn    │
              │  if user silent ~400ms AND a reply is ready:               │
              │     commit -> append to history -> diffusion.step(reply)   │
              │  else: diffusion.step("")  (blank keep-alive frame)        │
              └─────────────────────────────────────────────────────────┘
                                  │
                                  ▼ AVLatent (jpeg frame + tts mp3) ──> client

Client (browser)                      Server (this app)
┌──────────────────────┐   websocket  ┌─────────────────────────────────────┐
│ [avatar <img>]        │◄────────────│ Session                             │
│ [Start] [End]         │────────────►│  ├─ Orchestrator (dialogue v2)      │
│ mic ──► pcm chunks    │             │  ├─ ingress: ws bytes -> mic_in     │
│ frame + speech ◄──────│             │  └─ egress:  latents_out -> decode  │
└──────────────────────┘             │            -> ws [len|jpeg|mp3]      │
                                      └─────────────────────────────────────┘

What the stubs do

• —`DialogueDuplexModel` — faster-whisper tiny.en with word timestamps over a sliding 2s window every 200ms. New words are merged onto an absolute-timeline transcript using a commit cutoff, which de-duplicates the overlap between windows. Exposes tick(), is_user_active() (VAD), and the pending-user-turn accessors. An RMS energy gate suppresses silence so the ASR can't hallucinate.
• —`LLMResponder` — HF Inference Providers router (OpenAI-compatible). Tries Llama-3.3-70B-Instruct on Groq first (~250ms), falls back to 8B then 3B auto-route. Hard 1s per-call timeout; returns "" on timeout so a slow call just skips a commit.
• —`DialogueDiffusionModel` — renders the reply on a 640×360 frame (PIL) and speaks it with edge-tts (en-US-AndrewNeural). Idle ticks → blank frame, no audio.

Wire format

• —Client → server: raw int16 PCM @ 16kHz mono, no framing.
• —Server → client: [u32 LE video_len][JPEG bytes][mp3 bytes]. The client reads the length prefix, paints the JPEG to <img>, and plays the mp3 (if any) through a sequential AudioContext queue.

Config

The Space needs an HF_TOKEN secret (set on push) so LLMResponder can call the inference router. No other keys — edge-tts is free and unauthenticated.

Swapping in real models

orchestrator.py never imports the stubs — only app.py does. Replace DialogueDuplexModel / LLMResponder / DialogueDiffusionModel and make AVDecoder.decode turn real latent tensors into pixels + audio. The orchestrator, wire format, and client stay put.

Run locally

export HF_TOKEN=hf_...           # for the LLM router
pip install -r requirements.txt
uvicorn app:app --port 7860
# open http://localhost:7860  (mic needs HTTPS or localhost)