arnavk2610/silent-speech-vsr

Silent Speech — Visual Speech Recognition demo

Lip-reading from video alone (no audio). The Auto-AVSR Conformer model (mpc001/auto_avsr) fine-tuned to a single speaker with a rank-64 LoRA adapter — 21.9% WER on held-out data, vs 45.1% for the off-the-shelf model.

At build time the Space downloads the 1GB base checkpoint, merges the LoRA adapter (lora_r64_e5.pth) into it, and bakes the merged model into the image. app.py records a clip, runs mediapipe mouth-crop preprocessing (identical to training), and decodes with the model's beam search.

See the parent project's RESULTS.md for the full experiment write-up.