soapboxguy/MusicGen

1# AudioCraft objective metrics2 3In addition to training losses, AudioCraft provides a set of objective metrics4for audio synthesis and audio generation. As these metrics may require5extra dependencies and can be costly to train, they are often disabled by default.6This section provides guidance for setting up and using these metrics in7the AudioCraft training pipelines.8 9## Available metrics10 11### Audio synthesis quality metrics12 13#### SI-SNR14 15We provide an implementation of the Scale-Invariant Signal-to-Noise Ratio in PyTorch.16No specific requirement is needed for this metric. Please activate the metric at the17evaluation stage with the appropriate flag:18 19**Warning:** We report the opposite of the SI-SNR, e.g. multiplied by -1. This is due to internal 20    details where the SI-SNR score can also be used as a training loss function, where lower21    values should indicate better reconstruction. Negative values are such expected and a good sign! Those should be again multiplied by `-1` before publication :)22 23```shell24dora run <...> evaluate.metrics.sisnr=true25```26 27#### ViSQOL28 29We provide a Python wrapper around the ViSQOL [official implementation](https://github.com/google/visqol)30to conveniently run ViSQOL within the training pipelines.31 32One must specify the path to the ViSQOL installation through the configuration in order33to enable ViSQOL computations in AudioCraft:34 35```shell36# the first parameter is used to activate visqol computation while the second specify37# the path to visqol's library to be used by our python wrapper38dora run <...> evaluate.metrics.visqol=true metrics.visqol.bin=<path_to_visqol>39```40 41See an example grid: [Compression with ViSQOL](../audiocraft/grids/compression/encodec_musicgen_32khz.py)42 43To learn more about ViSQOL and how to build ViSQOL binary using bazel, please refer to the44instructions available in the [open source repository](https://github.com/google/visqol).45 46### Audio generation metrics47 48#### Frechet Audio Distance49 50Similarly to ViSQOL, we use a Python wrapper around the Frechet Audio Distance51[official implementation](https://github.com/google-research/google-research/tree/master/frechet_audio_distance)52in TensorFlow.53 54Note that we had to make several changes to the actual code in order to make it work.55Please refer to the [FrechetAudioDistanceMetric](../audiocraft/metrics/fad.py) class documentation56for more details. We do not plan to provide further support in obtaining a working setup for the57Frechet Audio Distance at this stage.58 59```shell60# the first parameter is used to activate FAD metric computation while the second specify61# the path to FAD library to be used by our python wrapper62dora run <...> evaluate.metrics.fad=true metrics.fad.bin=<path_to_google_research_repository>63```64 65See an example grid: [Evaluation with FAD](../audiocraft/grids/musicgen/musicgen_pretrained_32khz_eval.py)66 67#### Kullback-Leibler Divergence68 69We provide a PyTorch implementation of the Kullback-Leibler Divergence computed over the probabilities70of the labels obtained by a state-of-the-art audio classifier. We provide our implementation of the KLD71using the [PaSST classifier](https://github.com/kkoutini/PaSST).72 73In order to use the KLD metric over PaSST, you must install the PaSST library as an extra dependency:74```shell75pip install 'git+https://github.com/kkoutini/passt_hear21@0.0.19#egg=hear21passt'76```77 78Then similarly, you can use the metric activating the corresponding flag:79 80```shell81# one could extend the kld metric with additional audio classifier models that can then be picked through the configuration82dora run <...> evaluate.metrics.kld=true metrics.kld.model=passt83```84 85#### Text consistency86 87We provide a text-consistency metric, similarly to the MuLan Cycle Consistency from88[MusicLM](https://arxiv.org/pdf/2301.11325.pdf) or the CLAP score used in89[Make-An-Audio](https://arxiv.org/pdf/2301.12661v1.pdf).90More specifically, we provide a PyTorch implementation of a Text consistency metric91relying on a pre-trained [Contrastive Language-Audio Pretraining (CLAP)](https://github.com/LAION-AI/CLAP).92 93Please install the CLAP library as an extra dependency prior to using the metric:94```shell95pip install laion_clap96```97 98Then similarly, you can use the metric activating the corresponding flag:99 100```shell101# one could extend the text consistency metric with additional audio classifier models that can then be picked through the configuration102dora run ... evaluate.metrics.text_consistency=true metrics.text_consistency.model=clap103```104 105Note that the text consistency metric based on CLAP will require the CLAP checkpoint to be106provided in the configuration.107 108#### Chroma cosine similarity109 110Finally, as introduced in MusicGen, we provide a Chroma Cosine Similarity metric in PyTorch.111No specific requirement is needed for this metric. Please activate the metric at the112evaluation stage with the appropriate flag:113 114```shell115dora run ... evaluate.metrics.chroma_cosine=true116```117 118#### Comparing against reconstructed audio119 120For all the above audio generation metrics, we offer the option to compute the metric on the reconstructed audio121fed in EnCodec instead of the generated sample using the flag `<metric>.use_gt=true`.122 123## Example usage124 125You will find example of configuration for the different metrics introduced above in:126* The [musicgen's default solver](../config/solver/musicgen/default.yaml) for all audio generation metrics127* The [compression's default solver](../config/solver/compression/default.yaml) for all audio synthesis metrics128 129Similarly, we provide different examples in our grids:130* [Evaluation with ViSQOL](../audiocraft/grids/compression/encodec_musicgen_32khz.py)131* [Evaluation with FAD and others](../audiocraft/grids/musicgen/musicgen_pretrained_32khz_eval.py)132