Modelpublic

Joshua-1995/Singing-finetuned-DAC

sourceHugging Facecc-by-nc-4.0updated 3mo agoView on Hugging Face

1likes

Model Card

Singing-finetuned-DAC

Fine-tuned weights of the Descript Audio Codec (DAC) 24 kHz for singing voice.

No architecture changes. This is the official pretrained DAC 24 kHz model, further trained (full fine-tune) on ~472 h of (mostly monophonic) singing. The goal is better reconstruction of singing — especially high pitch range, vibrato, and F0 fidelity — which the original general-purpose DAC handles less well (it saw very little a-cappella singing during training).

Research / non-commercial use only. Code + reproduction: 👉 https://github.com/Joshua-1995/Singing-finetuned-DAC

Files

File	Size	Use
`dac_singing_finetune_24khz.pth`	286 MB	Inference — generator; `dac.DAC.load(...)`
`dac_singing_finetune_full_ckpt.tar.gz`	2.1 GB	Continue training — generator + discriminator + optimizer/scheduler

Results (pretrained DAC → fine-tuned)

Fixed held-out set of 160 singing clips across 6 datasets; identical clips before/after (metric definitions match the DAC paper's audiotools implementations).

Metric	Pretrained	Fine-tuned	Δ
Mel distance ↓	0.668	0.391	−0.277
STFT distance ↓	1.358	1.105	−0.253
SI-SDR (dB) ↑	−9.6	+15.6	+25.1
PESQ ↑	4.22	4.47	+0.25

Off-the-shelf DAC reconstructs singing waveforms poorly (SI-SDR −9.6 on singing vs ~16 dB reported on general audio); fine-tuning restores it to the codec's native quality regime on the singing domain. (scripts/eval_quality.py additionally reports STOI/MCD/F0.)

Usage

python

import dac
from audiotools import AudioSignal
model = dac.DAC.load("dac_singing_finetune_24khz.pth").eval().to("cuda")
sig = AudioSignal("song.wav").resample(24000).to_mono()
x = model.preprocess(sig.audio_data.cuda(), 24000)
z, codes, latents, _, _ = model.encode(x)   # z: (B, 1024, T) @ ~75 Hz
y = model.decode(z)

Variable bitrate (RVQ + quantizer dropout): 32 codebooks × 10 bits × 75 Hz → max 24 kbps; use fewer codebooks for lower rates. 24 kHz / 12 kHz bandwidth.

Training

Base: `weights_24khz_8kbps_0.0.4` (74.7 M generator, RVQ 32×1024 dim-8, hop 320 ≈ 75 Hz). Full fine-tune from the pretrained generator (the official release ships no discriminator, so MPD+MRD+MSD is re-initialized and warmed up). batch 16, 3 s segments, AdamW lr 1e-4, 200 k steps, quantizer_dropout 0.5. 1× NVIDIA RTX PRO 6000 (Blackwell), PyTorch 2.11 + CUDA 12.8.

Data (~472 h, 24 kHz mono, mostly monophonic singing)

Dataset	Lang	Hours	License / source
MSSV (Multi-Speaker Singing Voice)	KO	228.8	AI-Hub Terms of Use (#465) — Korea-only
GV (Guide Vocal)	KO	143.3	AI-Hub Terms of Use (#473) — Korea-only
ACE-KiSing	ZH	30.0	CC BY-NC 4.0
M4Singer	ZH	28.2	CC BY-NC-SA 4.0
HESD	KO	14.0	internal (not redistributed)
CSD	KO/EN	4.6	CC BY-NC-SA 4.0

This work used datasets from "The Open AI Dataset Project (AI-Hub, S. Korea)" (www.aihub.or.kr). MSSV/GV access is restricted to Korean nationals; overseas use requires a separate NIA agreement.

License

Research / non-commercial (CC BY-NC 4.0). The weights inherit the non-commercial terms of the training data. DAC code/architecture: MIT © Descript.

Acknowledgements

This work was supported by the GPU infrastructure provided by the Handong Global University AI Innovation Center. Training data includes AI-Hub datasets (see Data) and the Descript Audio Codec as the base model.

Citation

Built on the Descript Audio Codec:

bibtex

@inproceedings{kumar2023high,
  title={High-Fidelity Audio Compression with Improved {RVQGAN}},
  author={Kumar, Rithesh and Seetharaman, Prem and Luebs, Alejandro and Kumar, Ishaan and Kumar, Kundan},
  booktitle={NeurIPS}, year={2023}
}