NhanNguyen1309/audio-separation-api

1# AI Audio Stem Separation Backend2 3FastAPI backend for 2-stem and 4-stem separation with Demucs, plus separate4lightweight chord detection jobs with Chordino / NNLS Chroma when available.5 6## Local Setup7 8```bash9cd backend10python -m venv .venv11.venv\Scripts\activate12pip install -r requirements-dev.txt13uvicorn app.main:app --host 0.0.0.0 --port 786014```15 16FFmpeg and ffprobe must be installed and available on `PATH`.17Chord detection additionally needs Sonic Annotator and the Chordino / NNLS18Chroma Vamp plugin. The Docker image installs Sonic Annotator from the official19Linux release, builds the Chordino plugin from source, and verifies that20`vamp:nnls-chroma:chordino` is visible before the build can finish.21 22## API23 24- `GET /health`25- `POST /api/jobs` with multipart `file` and `stem_mode`26- `GET /api/jobs/{job_id}`27- `WS /ws/jobs/{job_id}`28- `GET /api/jobs/{job_id}/download`29- `DELETE /api/jobs/{job_id}`30- `POST /api/chords/jobs` with multipart `file`31- `GET /api/chords/jobs/{job_id}`32- `DELETE /api/chords/jobs/{job_id}`33 34No backend tempo, pitch, or transform endpoints are exposed. Chord detection is35backend-only and independent from Demucs separation. All editing preview and36final edited mix export happen locally in the frontend browser.37 38## Supported Inputs39 40- Formats: MP3, WAV, M4A41- Max duration: 300 seconds42- Max file size: 60MB43- Stem modes: `2-stem`, `4-stem`44- Chord detection max duration: 300 seconds45 46## Chord Smoothing47 48Chordino / NNLS Chroma returns raw chord changes that can flicker around melody49notes and transients. The backend now cleans those raw segments before returning50them to the frontend:51 521. detect beats with Sonic Annotator Vamp beat plugins when available532. fall back to optional `librosa`, then a 120 BPM beat grid543. snap chord boundaries to beats554. merge segments shorter than `CHORD_MIN_BEATS`565. smooth beat-level labels with a lightweight Viterbi pass576. limit each 4/4 bar to `CHORD_MAX_CHANGES_PER_BAR` chord runs by default587. merge duplicate consecutive chords and repair the final continuous timeline59 60Useful environment variables:61 62- `CHORD_MIN_BEATS=1.0`63- `CHORD_ENABLE_VITERBI=true`64- `CHORD_ENABLE_BAR_GUARDRAIL=true`65- `CHORD_MAX_CHANGES_PER_BAR=2`66- `CHORD_BEATS_PER_BAR=4`67 68The goal is a more musical timeline, not perfect music-theory correction.69Complex harmony and noisy mixes can still be simplified or misdetected.70 71## Hugging Face Spaces72 731. Create a new Hugging Face Space.742. Choose Docker as the Space SDK.753. Upload the contents of this `backend` folder.764. Set environment variables as needed:77   - `FRONTEND_ORIGIN=https://your-frontend-domain`78   - `DEMUCS_MODEL=htdemucs`79   - `DEMUCS_JOBS=2`80   - `DEMUCS_OVERLAP=0.05`81   - `DEMUCS_SHIFTS=0`82   - `PROCESSING_TIMEOUT_SECONDS=1200`83   - `MAX_DURATION_SECONDS=300`84   - `MAX_CHORD_DURATION_SECONDS=300`85   - `CHORD_MIN_BEATS=1.0`86   - `CHORD_ENABLE_VITERBI=true`87   - `CHORD_ENABLE_BAR_GUARDRAIL=true`88   - `CHORD_MAX_CHANGES_PER_BAR=2`89   - `CHORD_BEATS_PER_BAR=4`90   - `CHORD_JOBS_ROOT=/tmp/chord_jobs`915. Wait for the Docker build to complete.926. Test `https://your-space-name.hf.space/health`.937. Set the frontend env var:94   - `NEXT_PUBLIC_AUDIO_API_URL=https://your-space-name.hf.space`95 96The Docker build installs PyTorch support dependencies from PyPI first, then97installs the CPU PyTorch/TorchAudio wheels from the PyTorch CPU index with98`--no-deps`. Keep `torch` and `torchaudio` pinned together in99`requirements.txt`. This backend pins `torchaudio==2.7.1` because100`torchaudio==2.8.0` routes WAV saving through TorchCodec, which breaks Demucs101unless TorchCodec is also installed.102 103For faster CPU separation, the Docker defaults run Demucs with104`--overlap 0.05 --shifts 0`. Set `DEMUCS_SHIFTS=1` if you want a slightly more105robust but slower separation pass.106 107The Dockerfile installs a real chord runtime:108 109- downloads the static Sonic Annotator Linux binary110- builds `nnls-chroma.so` from the `c4dm/nnls-chroma` source repo111- copies the plugin into `/usr/local/lib/vamp`112- fails the build if `sonic-annotator -l` cannot see Chordino113 114Check `/health` for `chord_detection_available` and run:115 116```bash117sonic-annotator -l | grep chordino118python scripts/test_chords_api.py http://localhost:7860 ./song.wav119```120 121Do not use wildcard CORS in production. Set `FRONTEND_ORIGIN` to the frontend domain.122 123## Known MVP Limits124 125- Hugging Face free tier CPU processing may be slow.126- Jobs are stored in memory and are not multi-worker safe.127- Files are temporary and cleaned after roughly 1 hour.128- No auth, rate limiting, quotas, persistent object storage, or signed URLs yet.129 130## Production TODOs131 132- Add rate limiting and CAPTCHA.133- Add user accounts and paid quotas.134- Move jobs to Redis/Celery or another durable queue.135- Store outputs in Cloudflare R2 or another object store.136- Use signed download URLs.137- Add GPU execution on RunPod or a similar provider.138