justyoung/DiffSinger
1
1# DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism2[](https://arxiv.org/abs/2105.02446)3[](https://github.com/MoonInTheRiver/DiffSinger)4[](https://github.com/MoonInTheRiver/DiffSinger/releases)5 | [](https://huggingface.co/spaces/NATSpeech/DiffSpeech)6 | [English README](../README.md)7 8本仓库包含了我们的AAAI-2022 [论文](https://arxiv.org/abs/2105.02446)中提出的DiffSpeech (用于语音合成) 与 DiffSinger (用于歌声合成) 的官方Pytorch实现。9 10<table style="width:100%">11 <tr>12 <th>DiffSinger/DiffSpeech训练阶段</th>13 <th>DiffSinger/DiffSpeech推理阶段</th>14 </tr>15 <tr>16 <td><img src="resources/model_a.png" alt="Training" height="300"></td>17 <td><img src="resources/model_b.png" alt="Inference" height="300"></td>18 </tr>19</table>20 21:tada: :tada: :tada: **一些重要更新**:22- Mar.2, 2022: [MIDI-新版](README-SVS-opencpop-e2e.md): 重大更新 :sparkles:23 - Mar.1, 2022: [NeuralSVB](https://github.com/MoonInTheRiver/NeuralSVB), 为了歌声美化任务的代码，开源了 :sparkles: :sparkles: :sparkles: .24 - Feb.13, 2022: [NATSpeech](https://github.com/NATSpeech/NATSpeech), 一个升级后的代码框架, 包含了DiffSpeech和我们NeurIPS-2021的工作[PortaSpeech](https://openreview.net/forum?id=xmJsuh8xlq) 已经开源! :sparkles: :sparkles: :sparkles:. 25 - Jan.29, 2022: 支持了[MIDI-旧版](README-SVS-opencpop-cascade.md) 版本的歌声合成系统.26 - Jan.13, 2022: 支持了歌声合成系统, 开源了PopCS数据集.27 - Dec.19, 2021: 支持了语音合成系统. [HuggingFace🤗 Demo](https://huggingface.co/spaces/NATSpeech/DiffSpeech)28 29:rocket: **新闻**: 30 - Feb.24, 2022: 我们的新工作`NeuralSVB` 被 ACL-2022 接收 [](https://arxiv.org/abs/2202.13277). [音频演示](https://neuralsvb.github.io).31 - Dec.01, 2021: DiffSinger被AAAI-2022接收.32 - Sep.29, 2021: 我们的新工作`PortaSpeech: Portable and High-Quality Generative Text-to-Speech` 被NeurIPS-2021接收 [](https://arxiv.org/abs/2109.15166) .33 - May.06, 2021: 我们把这篇DiffSinger提交到了公开论文网站: Arxiv [](https://arxiv.org/abs/2105.02446).34 35## 安装依赖36```sh37conda create -n your_env_name python=3.838source activate your_env_name 39pip install -r requirements_2080.txt (GPU 2080Ti, CUDA 10.2)40or pip install -r requirements_3090.txt (GPU 3090, CUDA 11.4)41```42 43## DiffSpeech (语音合成的版本)44### 1. 准备工作45 46#### 数据准备47a) 下载并解压 [LJ Speech dataset](https://keithito.com/LJ-Speech-Dataset/), 创建软链接: `ln -s /xxx/LJSpeech-1.1/ data/raw/`48 49b) 下载并解压 [我们用MFA预处理好的对齐](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/mfa_outputs.tar): `tar -xvf mfa_outputs.tar; mv mfa_outputs data/processed/ljspeech/`50 51c) 按照如下脚本给数据集打包，打包后的二进制文件用于后续的训练和推理.52 53```sh54export PYTHONPATH=.55CUDA_VISIBLE_DEVICES=0 python data_gen/tts/bin/binarize.py --config configs/tts/lj/fs2.yaml56 57# `data/binary/ljspeech` will be generated.58```59 60#### 声码器准备61我们提供了[HifiGAN](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/0414_hifi_lj_1.zip)声码器的预训练模型.62请在训练声学模型前，先把声码器文件解压到`checkpoints`里。63 64### 2. 训练样例65 66首先你需要一个预训练好的FastSpeech2存档点. 你可以用[我们预训练好的模型](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/fs2_lj_1.zip), 或者跑下面这个指令从零开始训练FastSpeech2:67```sh68CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config configs/tts/lj/fs2.yaml --exp_name fs2_lj_1 --reset69```70然后为了训练DiffSpeech, 运行:71```sh72CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/lj_ds_beta6.yaml --exp_name lj_ds_beta6_1213 --reset73```74 75记得针对你的路径修改`usr/configs/lj_ds_beta6.yaml`里"fs2_ckpt"这个参数.76 77### 3. 推理样例78 79```sh80CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/lj_ds_beta6.yaml --exp_name lj_ds_beta6_1213 --reset --infer81```82 83我们也提供了:84 - [DiffSpeech](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/lj_ds_beta6_1213.zip)的预训练模型;85 - [FastSpeech 2](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/fs2_lj_1.zip)的预训练模型, 这是为了DiffSpeech里的浅扩散机制;86 87记得把预训练模型放在 `checkpoints` 目录.88 89## DiffSinger (歌声合成的版本)90 91### 0. 数据获取92- 见 [申请表](https://github.com/MoonInTheRiver/DiffSinger/blob/master/resources/apply_form.md).93- 数据集 [预览](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/popcs_preview.zip).94 95### 1. Preparation96#### 数据准备97a) 下载并解压PopCS, 创建软链接: `ln -s /xxx/popcs/ data/processed/popcs`98 99b) 按照如下脚本给数据集打包，打包后的二进制文件用于后续的训练和推理.100```sh101export PYTHONPATH=.102CUDA_VISIBLE_DEVICES=0 python data_gen/tts/bin/binarize.py --config usr/configs/popcs_ds_beta6.yaml103# `data/binary/popcs-pmf0` 会生成出来.104```105 106#### 声码器准备107我们提供了[HifiGAN-Singing](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/0109_hifigan_bigpopcs_hop128.zip)的预训练模型, 它专门为了歌声合成系统设计, 采用了NSF的技术。108请在训练声学模型前，先把声码器文件解压到`checkpoints`里。109 110(更新: 你也可以将我们提供的[训练更多步数的存档点](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/model_ckpt_steps_1512000.ckpt)放到声码器的文件夹里)111 112这个声码器是在大约70小时的较大数据集上训练的, 可以被认为是一个通用声码器。113 114### 2. 训练样例115首先你需要一个预训练好的FFT-Singer. 你可以用[我们预训练好的模型](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/popcs_fs2_pmf0_1230.zip), 或者用如下脚本从零训练FFT-Singer:116 117```sh118# First, train fft-singer;119CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_fs2.yaml --exp_name popcs_fs2_pmf0_1230 --reset120# Then, infer fft-singer;121CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_fs2.yaml --exp_name popcs_fs2_pmf0_1230 --reset --infer 122```123 124然后, 为了训练DiffSinger, 运行:125```sh126CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_ds_beta6_offline.yaml --exp_name popcs_ds_beta6_offline_pmf0_1230 --reset127```128 129记得针对你的路径修改`usr/configs/popcs_ds_beta6_offline.yaml`里"fs2_ckpt"这个参数.130 131### 3. 推理样例132```sh133CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_ds_beta6_offline.yaml --exp_name popcs_ds_beta6_offline_pmf0_1230 --reset --infer134```135 136我们也提供了:137 - [DiffSinger](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/popcs_ds_beta6_offline_pmf0_1230.zip)的预训练模型;138 - [FFT-Singer](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/popcs_fs2_pmf0_1230.zip)的预训练模型, 这是为了DiffSinger里的浅扩散机制;139 140记得把预训练模型放在 `checkpoints` 目录.141 142*请注意：*143 144-*我们原始论文中的PWG版本声码器已投入商业使用，因此我们提供此HifiGAN版本声码器作为替代品。*145 146-*我们这篇论文假设提供真实的F0来进行实验，如[1][2][3]等前作所做的那样，重点在频谱建模上，而非F0曲线的预测。如果你想对MIDI数据进行实验，从MIDI和歌词预测F0曲线（显式或隐式），请查看文档[MIDI-old-version](README-SVS-opencpop-cascade.md) 或 [MIDI-new-version](README-SVS-opencpop-e2e.md)。目前已经支持的MIDI数据集有: Opencpop*147 148[1] Adversarially trained multi-singer sequence-to-sequence singing synthesizer. Interspeech 2020.149 150[2] SEQUENCE-TO-SEQUENCE SINGING SYNTHESIS USING THE FEED-FORWARD TRANSFORMER. ICASSP 2020.151 152[3] DeepSinger : Singing Voice Synthesis with Data Mined From the Web. KDD 2020.153 154## Tensorboard155```sh156tensorboard --logdir_spec exp_name157```158<table style="width:100%">159 <tr>160 <td><img src="resources/tfb.png" alt="Tensorboard" height="250"></td>161 </tr>162</table>163 164## Mel 可视化165沿着纵轴, DiffSpeech: [0-80]; FastSpeech2: [80-160].166 167<table style="width:100%">168 <tr>169 <th>DiffSpeech vs. FastSpeech 2</th>170 </tr>171 <tr>172 <td><img src="resources/diffspeech-fs2.png" alt="DiffSpeech-vs-FastSpeech2" height="250"></td>173 </tr>174 <tr>175 <td><img src="resources/diffspeech-fs2-1.png" alt="DiffSpeech-vs-FastSpeech2" height="250"></td>176 </tr>177 <tr>178 <td><img src="resources/diffspeech-fs2-2.png" alt="DiffSpeech-vs-FastSpeech2" height="250"></td>179 </tr>180</table>181 182## Audio Demos183音频样本可以看我们的[样例页](https://diffsinger.github.io/).184 185我们也放了部分由DiffSpeech+HifiGAN (标记为[P]) 和 GTmel+HifiGAN (标记为[G]) 生成的测试集音频样例在：[resources/demos_1213](../resources/demos_1213). 186 187(对应这个预训练参数：[DiffSpeech](https://github.com/MoonInTheRiver/DiffSinger/releases/download/pretrain-model/lj_ds_beta6_1213.zip))188 189---190:rocket: :rocket: :rocket: **更新:**191 192新生成的歌声样例在：[resources/demos_0112](../resources/demos_0112).193 194## Citation195如果本仓库对你的研究和工作有用，请引用以下论文：196 197 @article{liu2021diffsinger,198 title={Diffsinger: Singing voice synthesis via shallow diffusion mechanism},199 author={Liu, Jinglin and Li, Chengxi and Ren, Yi and Chen, Feiyang and Liu, Peng and Zhao, Zhou},200 journal={arXiv preprint arXiv:2105.02446},201 volume={2},202 year={2021}}203 204 205## 鸣谢206我们的代码基于如下仓库:207* [denoising-diffusion-pytorch](https://github.com/lucidrains/denoising-diffusion-pytorch)208* [PyTorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning)209* [ParallelWaveGAN](https://github.com/kan-bayashi/ParallelWaveGAN)210* [HifiGAN](https://github.com/jik876/hifi-gan)211* [espnet](https://github.com/espnet/espnet)212* [DiffWave](https://github.com/lmnt-com/diffwave)