CoolFace
Apppublic

NhanNguyen1309/audio-separation-api

sourceHugging Faceupdated 4mo agoView on Hugging Face
0likes
App README

Audio Separation API

FastAPI backend for 2-stem and 4-stem audio separation using Demucs.

MDX-Net/audio-separator was tested as an experimental fast mode, but it was removed because it was slower than the stable Demucs pipeline in the current Hugging Face/free CPU environment.

This backend only validates uploads, separates stems, packages the ZIP result, serves downloads, runs lightweight chord detection jobs, and cleans temporary job files. Tempo/key preview, realtime tempo/pitch editing, and final edited mix export are frontend-only browser features.

Supported API surface:

  • —GET /health
  • —POST /api/jobs
  • —GET /api/jobs/{job_id}
  • —WS /ws/jobs/{job_id}
  • —GET /api/jobs/{job_id}/download
  • —DELETE /api/jobs/{job_id}
  • —POST /api/chords/jobs
  • —GET /api/chords/jobs/{job_id}
  • —DELETE /api/chords/jobs/{job_id}

Supported input limits:

  • —Input: MP3, WAV, M4A
  • —Max duration: 5 minutes
  • —Max file size: 60MB

Chord detection limits:

  • —Engine: Chordino / NNLS Chroma Vamp plugin
  • —Host: Sonic Annotator
  • —Endpoint: /api/chords/jobs
  • —Max duration: 5 minutes
  • —Input: MP3, WAV, M4A
  • —Output: corrected BPM, BPM candidates, beats, downbeats, bar offset, visual grid timing, chord timeline segments, and ensemble original-key analysis

Chord detection runs separately from Demucs separation. The Docker build now installs Sonic Annotator from the official Linux release and builds the NNLS Chroma / Chordino Vamp plugin from source. The build verifies the plugin with sonic-annotator -l, so a missing chord runtime fails during deployment instead of producing a half-enabled backend.

Chordino raw output can be noisy, so completed chord jobs post-process the timeline before returning it to the frontend:

  • —generate and score multiple BPM candidates with librosa onset features and autocorrelation
  • —prefer musically plausible lower-tempo candidates when double-time candidates have only marginally better onset scores
  • —compute beat times from the selected BPM
  • —estimate downbeats and barStartOffsetSec
  • —prefer optional madmom downbeats, then retain the existing librosa and fallback-grid beat paths
  • —prefer existing instrumental stems, combine existing bass + other stems, or build lightweight harmonic-enhanced audio before using the original mix
  • —snap chord boundaries forward to upcoming downbeats, backward to previous downbeats, then to nearby beats and visual cells; keep raw timestamps when no target is within the configured tolerance
  • —merge chord segments shorter than CHORD_MIN_BEATS
  • —run lightweight HMM/Viterbi smoothing on beat-level labels
  • —limit chord density per 4/4 bar with CHORD_MAX_CHANGES_PER_BAR
  • —return visualGrid.visualCellsPerBar=4, barDurationSec, and visualCellDurationSec

This reduces unrealistic rapid changes such as 3-4 chords inside one bar. It does not guarantee perfect harmony recognition; complex or noisy songs can still be simplified or misdetected.

Original-key detection is isolated from UI chord alignment. The UI uses the harmonic/aligned chord timeline, while key analysis keeps an original-mix, unsnapped Chordino timeline as its stable baseline. It evaluates all 24 major and minor keys with an explainable ensemble score that combines audio chroma, chord fit, tonic duration, first and last chords, cadence patterns, lightweight bass/root emphasis, and section-level voting. Experimental harmonic or snapped key timelines may only replace the original-mix baseline when enabled and when their confidence exceeds the baseline by KEY_CONFIDENCE_MARGIN.

The frontend should use the backend Smart Beat Grid metadata instead of recomputing timing from raw BPM alone. In 4/4, one visual cell equals one beat, so 4 visual cells equal one visual bar. Cell 5 is always bar 2 cell 1, starting at barStartOffsetSec + visualGrid.barDurationSec.

Chord post-processing defaults:

  • —CHORD_MIN_BEATS=1.0
  • —CHORD_ENABLE_VITERBI=true
  • —CHORD_ENABLE_BAR_GUARDRAIL=true
  • —CHORD_MAX_CHANGES_PER_BAR=2
  • —CHORD_BEATS_PER_BAR=4
  • —CHORD_USE_HARMONIC_SOURCE=true
  • —CHORD_ENABLE_SNAP=true
  • —CHORD_ENABLE_DOWNBEAT_SNAP=true
  • —CHORD_SNAP_TOLERANCE_SEC=0.2
  • —CHORD_FORWARD_DOWNBEAT_TOLERANCE_SEC=0.35
  • —CHORD_BACKWARD_DOWNBEAT_TOLERANCE_SEC=0.25
  • —CHORD_VISUAL_CELL_SNAP_TOLERANCE_SEC=0.25
  • —MADMOM_ENABLED=true
  • —KEY_USE_HARMONIC_CHORDS=false
  • —KEY_USE_SNAPPED_CHORDS=false
  • —KEY_CONFIDENCE_MARGIN=0.05

madmom is an optional runtime enhancement and is not installed as a hard dependency. When it is unavailable or returns an invalid beat grid, chord analysis continues with the existing librosa tracker and fallback grid. The completed chord-job response includes chordAudioSource, beatSource, snapConfig, keyAnalysisChords, keyAudioSource, keyDetectionSource, and warning codes so callers can inspect the selected path. Rhythm analysis and original-key chroma extraction continue to use the prepared original mix.

Runtime Demucs defaults:

  • —DEMUCS_JOBS=2
  • —DEMUCS_OVERLAP=0.05
  • —DEMUCS_SHIFTS=0

These produce a faster CPU path. Use DEMUCS_SHIFTS=1 if you want a slightly slower but more stable separation pass.

The Docker image pins torch==2.7.1 and torchaudio==2.7.1 for Demucs. Avoid upgrading TorchAudio to 2.8.0 unless TorchCodec is installed and tested, because 2.8.0 can fail while Demucs writes WAV stems.

Verify inside the container:

bash
sonic-annotator -l | grep chordino

Smoke test a deployed chord endpoint:

bash
python scripts/test_chords_api.py https://your-space-name.hf.space ./song.wav

Known chord detection limitations: complex chords may be wrong, noisy mixes reduce accuracy, free CPU can be slow, and plugin installation differs across Linux images.