ReddyvariSreenath/regionalai-video

RegionalAI Video

Create one video. Reach India in every language.

Upload a source video, pick target languages (Hindi, Telugu, Kannada, Tamil, Malayalam, Bengali, Marathi), and get back synchronized dubbed videos, translated audio tracks, and subtitles for each language.

Performance

Segments within a language, and languages within a project, process concurrently (capped at 3 languages and 2-6 TTS calls at once, depending on the voice engine) rather than one step at a time. Measured on a ~57-second, 19-segment test video, 2 target languages, on this machine: 88s sequential → 60s parallel (~32% faster). There's no hard duration limit enforced, but nothing beyond a ~1-minute clip has been tested yet — a 10-20 minute video hasn't been benchmarked and may still take a while, since transcription itself (one CPU pass over the whole video) doesn't parallelize.

How it works

Upload video
   -> Extract audio (ffmpeg)
   -> Transcribe (faster-whisper)
   -> Rewrite into casual, spoken-style English (local LLM via Ollama)
   -> Translate transcript per target language (Google Translate)
   -> Generate subtitles (.srt)
   -> Generate regional voice (gTTS) synced to original segment timing
   -> Mux dubbed audio back onto original video (ffmpeg)
   -> Download audio / subtitles / dubbed video per language

Stack

• —Backend: FastAPI + SQLAlchemy (SQLite)
• —Transcription: faster-whisper (local, CPU)
• —Casual tone rewrite: local LLM via Ollama (llama3.2-local) — rewrites the English transcript into natural spoken phrasing before translation, so dubbed output sounds conversational instead of like a formal script read aloud. English-only: the 3B model isn't reliable enough to translate directly into most target Indian languages (confirmed inaccurate for Telugu), so it never touches non-English text — actual translation is still Google Translate for every language. Toggle with CASUALIZE_TRANSCRIPT=false; safely no-ops if Ollama isn't running.
• —Translation: Google Translate via deep-translator (swappable)
• —Voice: two selectable engines so output doesn't always sound identical — gtts (Google Translate TTS, default, faster) and mms (Meta's MMS-TTS neural models, run locally via transformers+torch, genuinely different voice, models download once from Hugging Face and cache on disk). Both free, no API key. Picked per project via voice_variant in /process or the "Voice style" radio in the demo UI. Neither offers multiple voices within a language yet — that needs a paid TTS API (Azure/Google Cloud/ElevenLabs); this just avoids every video sounding the same.
• —Video/audio processing: ffmpeg

Requirements

• —Python 3.11+
• —ffmpeg and ffprobe on PATH
• —Ollama running locally with a llama3.2-local model, for the casual-tone rewrite (optional — pipeline still works without it, just skips that step)

Setup

cd backend
python -m venv .venv
.venv\Scripts\activate        # Windows
pip install torch --index-url https://download.pytorch.org/whl/cpu   # CPU-only wheel; plain PyPI pulls a multi-GB CUDA build
pip install -r requirements.txt
copy .env.example .env

Run the API

cd backend
.venv\Scripts\activate
uvicorn app.main:app --reload --port 8921

Open http://127.0.0.1:8921/docs for interactive API docs.

Run the demo UI

A Streamlit app that gives the real end-user flow: pick a video file, pick source/target languages, click Start Dubbing, watch per-language progress, then preview and download each dubbed video/audio/subtitle. Needs the API server running (above) at the same time.

cd backend
.venv\Scripts\activate
pip install -r ../frontend/requirements.txt   # first time only
streamlit run ../frontend/streamlit_app.py --server.port 8922

Open http://127.0.0.1:8922

Ports 8921 (API) and 8922 (UI) are dedicated to this project so they don't collide with other projects you run in parallel. If the UI ever needs to point at a different API port, set API_HOST before launching it, e.g. set API_HOST=http://127.0.0.1:8921.

API flow

1. 1.POST /api/projects — create a project (name, source_language)
2. 2.POST /api/projects/{id}/upload — upload the source video file
3. 3.POST /api/projects/{id}/process — start processing, body: {"languages": ["hi", "te", "kn"], "voice_variant": "gtts"} (voice_variant optional, gtts or mms, defaults to gtts)
4. 4.GET /api/projects/{id}/status — poll progress per language
5. 5.GET /api/projects/{id}/outputs — list download links once complete
6. 6.GET /api/projects/{id}/download/{language}/{audio|subtitles|video} — download a file

Run the pipeline directly (no API/DB)

python scripts/run_pipeline.py sample_input/sample_source_video.mp4 --source en --targets hi te kn

Outputs land in storage/outputs/<project-id>/<language>/.

Project structure

backend/
  app/
    main.py            FastAPI app
    config.py           settings
    database.py          SQLAlchemy setup
    api/                 routes (projects, jobs)
    models/               Project, Job
    services/
      transcription.py    faster-whisper
      casualize.py          casual-tone rewrite via local LLM (Ollama)
      translation.py       deep-translator
      subtitles.py          .srt generation
      voice.py                gTTS + timing sync
      video.py                  ffmpeg wrapper
      pipeline.py                orchestrator
    worker.py            isolated subprocess entrypoint for the pipeline
  requirements.txt
frontend/
  streamlit_app.py      demo UI (upload, pick languages, watch progress, download)
  requirements.txt
scripts/
  run_pipeline.py       local CLI runner
storage/
  uploads/               uploaded source videos
  outputs/                 generated per-language outputs

Deploy (Hugging Face Spaces)

Chosen over Streamlit Community Cloud (can't host the separate FastAPI backend — single-service only) and Render (free tier RAM is too tight for faster-whisper + torch/transformers running together, plus it spins down after 15 min idle). HF Spaces' free Docker tier gives 2 vCPU / 16GB RAM, which comfortably fits this stack.

The Dockerfile runs both services in one container: FastAPI internally on 8921, Streamlit on 7860 (the port Spaces expects) as the public app, talking to the API over localhost.

Known limits of a free-tier deploy (not deployment bugs — inherent to a free/ephemeral host, worth knowing before pointing real users at it):

• —Storage is ephemeral — every restart/redeploy wipes uploaded videos, outputs, and the SQLite DB. Fine for "try it, download your video" use; no persistent project history until real cloud storage is added.
• —The casual-tone rewrite (Ollama) is disabled by default in the container (CASUALIZE_TRANSCRIPT=false) — no free host can run a 2GB local LLM as a background service. Dubbing still works, just without that phrasing pass.
• —Single shared CPU instance, no autoscaling — fine for early free-tier testing, not for real paid-scale concurrent usage.

Steps

1. 1.Create a new Space at huggingface.co/new-space — pick Docker as the SDK, any name, public or private visibility as you prefer.
2. 2.Hugging Face gives you a git remote URL for the Space (https://huggingface.co/spaces/<you>/<space-name>). Add it as a remote here and push:

   git remote add space https://huggingface.co/spaces/<you>/<space-name>
   git push space master:main

1. 1.The Space builds the Dockerfile automatically (first build takes several minutes — torch/transformers/faster-whisper are large). Watch progress in the Space's "Logs" tab.
2. 2.Once built, the Space serves the Streamlit UI directly at its public URL.

This pushes to Hugging Face, not GitHub — separate from this project's "don't push to GitHub" rule, and only happens when you explicitly run the commands above yourself.

Roadmap

Phase 1 (this MVP): local engine + FastAPI, one video in, multiple dubbed videos out. ✓ Phase 2: professional UI ✓, voice variety ✓, speed ✓, free-tier public deploy (this section). Next: auth, usage limits/credits, payments (₹249/5 videos etc.), real cloud storage for project history.