ZeroTwo3/videoshop-backend

1# AudioCraft objective metrics2 3In addition to training losses, AudioCraft provides a set of objective metrics4for audio synthesis and audio generation. As these metrics may require5extra dependencies and can be costly to train, they are often disabled by default.6This section provides guidance for setting up and using these metrics in7the AudioCraft training pipelines.8 9## Available metrics10 11### Audio synthesis quality metrics12 13#### SI-SNR14 15We provide an implementation of the Scale-Invariant Signal-to-Noise Ratio in PyTorch.16No specific requirement is needed for this metric. Please activate the metric at the17evaluation stage with the appropriate flag:18 19```shell20dora run <...> evaluate.metrics.sisnr=true21```22 23#### ViSQOL24 25We provide a Python wrapper around the ViSQOL [official implementation](https://github.com/google/visqol)26to conveniently run ViSQOL within the training pipelines.27 28One must specify the path to the ViSQOL installation through the configuration in order29to enable ViSQOL computations in AudioCraft:30 31```shell32# the first parameter is used to activate visqol computation while the second specify33# the path to visqol's library to be used by our python wrapper34dora run <...> evaluate.metrics.visqol=true metrics.visqol.bin=<path_to_visqol>35```36 37See an example grid: [Compression with ViSQOL](../audiocraft/grids/compression/encodec_musicgen_32khz.py)38 39To learn more about ViSQOL and how to build ViSQOL binary using bazel, please refer to the40instructions available in the [open source repository](https://github.com/google/visqol).41 42### Audio generation metrics43 44#### Frechet Audio Distance45 46Similarly to ViSQOL, we use a Python wrapper around the Frechet Audio Distance47[official implementation](https://github.com/google-research/google-research/tree/master/frechet_audio_distance)48in TensorFlow.49 50Note that we had to make several changes to the actual code in order to make it work.51Please refer to the [FrechetAudioDistanceMetric](../audiocraft/metrics/fad.py) class documentation52for more details. We do not plan to provide further support in obtaining a working setup for the53Frechet Audio Distance at this stage.54 55```shell56# the first parameter is used to activate FAD metric computation while the second specify57# the path to FAD library to be used by our python wrapper58dora run <...> evaluate.metrics.fad=true metrics.fad.bin=<path_to_google_research_repository>59```60 61See an example grid: [Evaluation with FAD](../audiocraft/grids/musicgen/musicgen_pretrained_32khz_eval.py)62 63#### Kullback-Leibler Divergence64 65We provide a PyTorch implementation of the Kullback-Leibler Divergence computed over the probabilities66of the labels obtained by a state-of-the-art audio classifier. We provide our implementation of the KLD67using the [PaSST classifier](https://github.com/kkoutini/PaSST).68 69In order to use the KLD metric over PaSST, you must install the PaSST library as an extra dependency:70```shell71pip install 'git+https://github.com/kkoutini/passt_hear21@0.0.19#egg=hear21passt'72```73 74Then similarly, you can use the metric activating the corresponding flag:75 76```shell77# one could extend the kld metric with additional audio classifier models that can then be picked through the configuration78dora run <...> evaluate.metrics.kld=true metrics.kld.model=passt79```80 81#### Text consistency82 83We provide a text-consistency metric, similarly to the MuLan Cycle Consistency from84[MusicLM](https://arxiv.org/pdf/2301.11325.pdf) or the CLAP score used in85[Make-An-Audio](https://arxiv.org/pdf/2301.12661v1.pdf).86More specifically, we provide a PyTorch implementation of a Text consistency metric87relying on a pre-trained [Contrastive Language-Audio Pretraining (CLAP)](https://github.com/LAION-AI/CLAP).88 89Please install the CLAP library as an extra dependency prior to using the metric:90```shell91pip install laion_clap92```93 94Then similarly, you can use the metric activating the corresponding flag:95 96```shell97# one could extend the text consistency metric with additional audio classifier models that can then be picked through the configuration98dora run ... evaluate.metrics.text_consistency=true metrics.text_consistency.model=clap99```100 101Note that the text consistency metric based on CLAP will require the CLAP checkpoint to be102provided in the configuration.103 104#### Chroma cosine similarity105 106Finally, as introduced in MusicGen, we provide a Chroma Cosine Similarity metric in PyTorch.107No specific requirement is needed for this metric. Please activate the metric at the108evaluation stage with the appropriate flag:109 110```shell111dora run ... evaluate.metrics.chroma_cosine=true112```113 114#### Comparing against reconstructed audio115 116For all the above audio generation metrics, we offer the option to compute the metric on the reconstructed audio117fed in EnCodec instead of the generated sample using the flag `<metric>.use_gt=true`.118 119## Example usage120 121You will find example of configuration for the different metrics introduced above in:122* The [musicgen's default solver](../config/solver/musicgen/default.yaml) for all audio generation metrics123* The [compression's default solver](../config/solver/compression/default.yaml) for all audio synthesis metrics124 125Similarly, we provide different examples in our grids:126* [Evaluation with ViSQOL](../audiocraft/grids/compression/encodec_musicgen_32khz.py)127* [Evaluation with FAD and others](../audiocraft/grids/musicgen/musicgen_pretrained_32khz_eval.py)128