hugging-apps/diffsynth-music-demo

DiffSynth-Music

Demo of **DiffSynth-Studio/DiffSynth-Music** — Audio-Conditioned KV-Cache Adapters for Controllable Music Generation (arXiv:2609.12774).

DiffSynth-Music keeps the ACE-Step-1.5-XL backbone frozen and attaches Diffusion-Template adapters that encode a control audio track into a layer-wise KV cache, injected into the DiT's attention. Five control modes are exposed here:

The "Make a control signal from a full song" panel runs Demucs track separation (and, for Prosody, pitch/envelope extraction) so you can go from a complete song to a usable control signal in one click.

Notes

• —Runs on ZeroGPU: ~36 GB of weights (backbone DiT + all three ≈8 B template models, plus VAE, conditioner, text encoder and Demucs) are resident together. Generation is quick — roughly 10 s for a 60 s track at 50 steps, 43 s for 150 s at 100 steps.
• —Control audio is cropped to the loudest window of the requested length rather than to its first N seconds. Songs commonly open with a quiet intro (the authors' own isolated vocal stem sits at −49 dBFS over its first 20 s), and because fuse_track balances the render down to the control track's RMS, cropping from the start would scale short renders into near-silence.
• —Reference mode analyses the loudest ~10 s window of the control audio; the input is cropped to 60 s first to keep the scan fast.
• —Prosody extraction follows the reference implementation, with the pYIN pitch track estimated on a 16 kHz down-sample for speed (envelope and carrier stay at 48 kHz).

Credits

Example control clips in examples/ are the authors' own demo assets from the DiffSynth-Music model repository (Apache-2.0).