justyoung/DiffSinger
1
1# DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism2[](https://arxiv.org/abs/2105.02446)3[](https://github.com/MoonInTheRiver/DiffSinger)4[](https://github.com/MoonInTheRiver/DiffSinger/releases)5 | [Interactive🤗 SVS](https://huggingface.co/spaces/Silentlin/DiffSinger)6 7## DiffSinger (SVS)8 9### PART1. [Run DiffSinger on PopCS](README-SVS-popcs.md)10In PART1, we only focus on spectrum modeling (acoustic model) and assume the ground-truth (GT) F0 to be given as the pitch information following these papers [1][2][3]. If you want to conduct experiments with F0 prediction, please move to PART2.11 12Thus, the pipeline of this part can be summarized as:13 14```15[lyrics] -> [linguistic representation] (Frontend)16[linguistic representation] + [GT F0] + [GT phoneme duration] -> [mel-spectrogram] (Acoustic model)17[mel-spectrogram] + [GT F0] -> [waveform] (Vocoder)18```19 20 21[1] Adversarially trained multi-singer sequence-to-sequence singing synthesizer. Interspeech 2020.22 23[2] SEQUENCE-TO-SEQUENCE SINGING SYNTHESIS USING THE FEED-FORWARD TRANSFORMER. ICASSP 2020.24 25[3] DeepSinger : Singing Voice Synthesis with Data Mined From the Web. KDD 2020.26 27Click here for detailed instructions: [link](README-SVS-popcs.md).28 29 30### PART2. [Run DiffSinger on Opencpop](README-SVS-opencpop-cascade.md)31Thanks [Opencpop team](https://wenet.org.cn/opencpop/) for releasing their SVS dataset with MIDI label, **Jan.20, 2022** (after we published our paper).32 33Since there are elaborately annotated MIDI labels, we are able to supplement the pipeline in PART 1 by adding a naive melody frontend.34 35#### 2.A36Thus, the pipeline of [2.A](README-SVS-opencpop-cascade.md) can be summarized as:37 38```39[lyrics] + [MIDI] -> [linguistic representation (with MIDI information)] + [predicted F0] + [predicted phoneme duration] (Melody frontend)40[linguistic representation] + [predicted F0] + [predicted phoneme duration] -> [mel-spectrogram] (Acoustic model)41[mel-spectrogram] + [predicted F0] -> [waveform] (Vocoder)42```43 44Click here for detailed instructions: [link](README-SVS-opencpop-cascade.md).45 46#### 2.B47In 2.1, we find that if we predict F0 explicitly in the melody frontend, there will be many bad cases of uv/v prediction. Then, we abandon the explicit prediction of the F0 curve in the melody frontend and make a joint prediction with spectrograms.48 49Thus, the pipeline of [2.B](README-SVS-opencpop-e2e.md) can be summarized as:50```51[lyrics] + [MIDI] -> [linguistic representation] + [predicted phoneme duration] (Melody frontend)52[linguistic representation (with MIDI information)] + [predicted phoneme duration] -> [mel-spectrogram] (Acoustic model)53[mel-spectrogram] -> [predicted F0] (Pitch extractor)54[mel-spectrogram] + [predicted F0] -> [waveform] (Vocoder)55```56 57Click here for detailed instructions: [link](README-SVS-opencpop-e2e.md).58 59### FAQ60Q1: Why do I need F0 in Vocoders?61 62A1: See vocoder parts in HiFiSinger, DiffSinger or SingGAN. This is a common practice now.63 64Q2: Why not run MIDI version SVS on PopCS dataset? or Why not release MIDI labels for PopCS dataset?65 66A2: Our laboratory has no funds to label PopCS dataset. But there are funds for labeling other singing dataset, which is coming soon.67 68Q3: Why " 'HifiGAN' object has no attribute 'model' "?69 70A3: Please put the pretrained vocoders in your `checkpoints` dictionary.71 72Q4: How to check whether I use GT information or predicted information during inference from packed test set?73 74A4: Please see codes [here](https://github.com/MoonInTheRiver/DiffSinger/blob/55e2f46068af6e69940a9f8f02d306c24a940cab/tasks/tts/fs2.py#L343).75 76...