Aluode/PerceptionLabPortable

1# coding=utf-82# Copyright 2023 The Fairseq Authors, Microsoft Research, and the HuggingFace Inc. team. All rights reserved.3#4# Licensed under the Apache License, Version 2.0 (the "License");5# you may not use this file except in compliance with the License.6# You may obtain a copy of the License at7#8#     http://www.apache.org/licenses/LICENSE-2.09#10# Unless required by applicable law or agreed to in writing, software11# distributed under the License is distributed on an "AS IS" BASIS,12# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.13# See the License for the specific language governing permissions and14# limitations under the License.15"""SpeechT5 model configuration"""16 17import functools18import operator19 20from ...configuration_utils import PretrainedConfig21from ...utils import logging22 23 24logger = logging.get_logger(__name__)25 26 27class SpeechT5Config(PretrainedConfig):28    r"""29    This is the configuration class to store the configuration of a [`SpeechT5Model`]. It is used to instantiate a30    SpeechT5 model according to the specified arguments, defining the model architecture. Instantiating a configuration31    with the defaults will yield a similar configuration to that of the SpeechT532    [microsoft/speecht5_asr](https://huggingface.co/microsoft/speecht5_asr) architecture.33 34    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the35    documentation from [`PretrainedConfig`] for more information.36 37    Args:38        vocab_size (`int`, *optional*, defaults to 81):39            Vocabulary size of the SpeechT5 model. Defines the number of different tokens that can be represented by40            the `inputs_ids` passed to the forward method of [`SpeechT5Model`].41        hidden_size (`int`, *optional*, defaults to 768):42            Dimensionality of the encoder layers and the pooler layer.43        encoder_layers (`int`, *optional*, defaults to 12):44            Number of hidden layers in the Transformer encoder.45        encoder_attention_heads (`int`, *optional*, defaults to 12):46            Number of attention heads for each attention layer in the Transformer encoder.47        encoder_ffn_dim (`int`, *optional*, defaults to 3072):48            Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.49        encoder_layerdrop (`float`, *optional*, defaults to 0.1):50            The LayerDrop probability for the encoder. See the [LayerDrop paper](see https://huggingface.co/papers/1909.11556)51            for more details.52        decoder_layers (`int`, *optional*, defaults to 6):53            Number of hidden layers in the Transformer decoder.54        decoder_attention_heads (`int`, *optional*, defaults to 12):55            Number of attention heads for each attention layer in the Transformer decoder.56        decoder_ffn_dim (`int`, *optional*, defaults to 3072):57            Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer decoder.58        decoder_layerdrop (`float`, *optional*, defaults to 0.1):59            The LayerDrop probability for the decoder. See the [LayerDrop paper](see https://huggingface.co/papers/1909.11556)60            for more details.61        hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`):62            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,63            `"relu"`, `"selu"` and `"gelu_new"` are supported.64        positional_dropout (`float`, *optional*, defaults to 0.1):65            The dropout probability for the text position encoding layers.66        hidden_dropout (`float`, *optional*, defaults to 0.1):67            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.68        attention_dropout (`float`, *optional*, defaults to 0.1):69            The dropout ratio for the attention probabilities.70        activation_dropout (`float`, *optional*, defaults to 0.1):71            The dropout ratio for activations inside the fully connected layer.72        initializer_range (`float`, *optional*, defaults to 0.02):73            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.74        layer_norm_eps (`float`, *optional*, defaults to 1e-5):75            The epsilon used by the layer normalization layers.76        scale_embedding (`bool`, *optional*, defaults to `False`):77            Scale embeddings by diving by sqrt(d_model).78        feat_extract_norm (`str`, *optional*, defaults to `"group"`):79            The norm to be applied to 1D convolutional layers in the speech encoder pre-net. One of `"group"` for group80            normalization of only the first 1D convolutional layer or `"layer"` for layer normalization of all 1D81            convolutional layers.82        feat_proj_dropout (`float`, *optional*, defaults to 0.0):83            The dropout probability for output of the speech encoder pre-net.84        feat_extract_activation (`str, `optional`, defaults to `"gelu"`):85            The non-linear activation function (function or string) in the 1D convolutional layers of the feature86            extractor. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported.87        conv_dim (`tuple[int]` or `list[int]`, *optional*, defaults to `(512, 512, 512, 512, 512, 512, 512)`):88            A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the89            speech encoder pre-net. The length of *conv_dim* defines the number of 1D convolutional layers.90        conv_stride (`tuple[int]` or `list[int]`, *optional*, defaults to `(5, 2, 2, 2, 2, 2, 2)`):91            A tuple of integers defining the stride of each 1D convolutional layer in the speech encoder pre-net. The92            length of *conv_stride* defines the number of convolutional layers and has to match the length of93            *conv_dim*.94        conv_kernel (`tuple[int]` or `list[int]`, *optional*, defaults to `(10, 3, 3, 3, 3, 3, 3)`):95            A tuple of integers defining the kernel size of each 1D convolutional layer in the speech encoder pre-net.96            The length of *conv_kernel* defines the number of convolutional layers and has to match the length of97            *conv_dim*.98        conv_bias (`bool`, *optional*, defaults to `False`):99            Whether the 1D convolutional layers have a bias.100        num_conv_pos_embeddings (`int`, *optional*, defaults to 128):101            Number of convolutional positional embeddings. Defines the kernel size of 1D convolutional positional102            embeddings layer.103        num_conv_pos_embedding_groups (`int`, *optional*, defaults to 16):104            Number of groups of 1D convolutional positional embeddings layer.105        apply_spec_augment (`bool`, *optional*, defaults to `True`):106            Whether to apply *SpecAugment* data augmentation to the outputs of the speech encoder pre-net. For107            reference see [SpecAugment: A Simple Data Augmentation Method for Automatic Speech108            Recognition](https://huggingface.co/papers/1904.08779).109        mask_time_prob (`float`, *optional*, defaults to 0.05):110            Percentage (between 0 and 1) of all feature vectors along the time axis which will be masked. The masking111            procedure generates ''mask_time_prob*len(time_axis)/mask_time_length'' independent masks over the axis. If112            reasoning from the probability of each feature vector to be chosen as the start of the vector span to be113            masked, *mask_time_prob* should be `prob_vector_start*mask_time_length`. Note that overlap may decrease the114            actual percentage of masked vectors. This is only relevant if `apply_spec_augment is True`.115        mask_time_length (`int`, *optional*, defaults to 10):116            Length of vector span along the time axis.117        mask_time_min_masks (`int`, *optional*, defaults to 2),:118            The minimum number of masks of length `mask_feature_length` generated along the time axis, each time step,119            irrespectively of `mask_feature_prob`. Only relevant if ''mask_time_prob*len(time_axis)/mask_time_length <120            mask_time_min_masks''121        mask_feature_prob (`float`, *optional*, defaults to 0.0):122            Percentage (between 0 and 1) of all feature vectors along the feature axis which will be masked. The123            masking procedure generates ''mask_feature_prob*len(feature_axis)/mask_time_length'' independent masks over124            the axis. If reasoning from the probability of each feature vector to be chosen as the start of the vector125            span to be masked, *mask_feature_prob* should be `prob_vector_start*mask_feature_length`. Note that overlap126            may decrease the actual percentage of masked vectors. This is only relevant if `apply_spec_augment is127            True`.128        mask_feature_length (`int`, *optional*, defaults to 10):129            Length of vector span along the feature axis.130        mask_feature_min_masks (`int`, *optional*, defaults to 0),:131            The minimum number of masks of length `mask_feature_length` generated along the feature axis, each time132            step, irrespectively of `mask_feature_prob`. Only relevant if133            ''mask_feature_prob*len(feature_axis)/mask_feature_length < mask_feature_min_masks''134        num_mel_bins (`int`, *optional*, defaults to 80):135            Number of mel features used per input features. Used by the speech decoder pre-net. Should correspond to136            the value used in the [`SpeechT5Processor`] class.137        speech_decoder_prenet_layers (`int`, *optional*, defaults to 2):138            Number of layers in the speech decoder pre-net.139        speech_decoder_prenet_units (`int`, *optional*, defaults to 256):140            Dimensionality of the layers in the speech decoder pre-net.141        speech_decoder_prenet_dropout (`float`, *optional*, defaults to 0.5):142            The dropout probability for the speech decoder pre-net layers.143        speaker_embedding_dim (`int`, *optional*, defaults to 512):144            Dimensionality of the *XVector* embedding vectors.145        speech_decoder_postnet_layers (`int`, *optional*, defaults to 5):146            Number of layers in the speech decoder post-net.147        speech_decoder_postnet_units (`int`, *optional*, defaults to 256):148            Dimensionality of the layers in the speech decoder post-net.149        speech_decoder_postnet_kernel (`int`, *optional*, defaults to 5):150            Number of convolutional filter channels in the speech decoder post-net.151        speech_decoder_postnet_dropout (`float`, *optional*, defaults to 0.5):152            The dropout probability for the speech decoder post-net layers.153        reduction_factor (`int`, *optional*, defaults to 2):154            Spectrogram length reduction factor for the speech decoder inputs.155        max_speech_positions (`int`, *optional*, defaults to 4000):156            The maximum sequence length of speech features that this model might ever be used with.157        max_text_positions (`int`, *optional*, defaults to 450):158            The maximum sequence length of text features that this model might ever be used with.159        encoder_max_relative_position (`int`, *optional*, defaults to 160):160            Maximum distance for relative position embedding in the encoder.161        use_guided_attention_loss (`bool`, *optional*, defaults to `True`):162            Whether to apply guided attention loss while training the TTS model.163        guided_attention_loss_num_heads (`int`, *optional*, defaults to 2):164            Number of attention heads the guided attention loss will be applied to. Use -1 to apply this loss to all165            attention heads.166        guided_attention_loss_sigma (`float`, *optional*, defaults to 0.4):167            Standard deviation for guided attention loss.168        guided_attention_loss_scale (`float`, *optional*, defaults to 10.0):169            Scaling coefficient for guided attention loss (also known as lambda).170        use_cache (`bool`, *optional*, defaults to `True`):171            Whether or not the model should return the last key/values attentions (not used by all models).172 173    Example:174 175    ```python176    >>> from transformers import SpeechT5Model, SpeechT5Config177 178    >>> # Initializing a "microsoft/speecht5_asr" style configuration179    >>> configuration = SpeechT5Config()180 181    >>> # Initializing a model (with random weights) from the "microsoft/speecht5_asr" style configuration182    >>> model = SpeechT5Model(configuration)183 184    >>> # Accessing the model configuration185    >>> configuration = model.config186    ```"""187 188    model_type = "speecht5"189    attribute_map = {"num_attention_heads": "encoder_attention_heads", "num_hidden_layers": "encoder_layers"}190 191    def __init__(192        self,193        vocab_size=81,194        hidden_size=768,195        encoder_layers=12,196        encoder_attention_heads=12,197        encoder_ffn_dim=3072,198        encoder_layerdrop=0.1,199        decoder_layers=6,200        decoder_ffn_dim=3072,201        decoder_attention_heads=12,202        decoder_layerdrop=0.1,203        hidden_act="gelu",204        positional_dropout=0.1,205        hidden_dropout=0.1,206        attention_dropout=0.1,207        activation_dropout=0.1,208        initializer_range=0.02,209        layer_norm_eps=1e-5,210        scale_embedding=False,211        feat_extract_norm="group",212        feat_proj_dropout=0.0,213        feat_extract_activation="gelu",214        conv_dim=(512, 512, 512, 512, 512, 512, 512),215        conv_stride=(5, 2, 2, 2, 2, 2, 2),216        conv_kernel=(10, 3, 3, 3, 3, 2, 2),217        conv_bias=False,218        num_conv_pos_embeddings=128,219        num_conv_pos_embedding_groups=16,220        apply_spec_augment=True,221        mask_time_prob=0.05,222        mask_time_length=10,223        mask_time_min_masks=2,224        mask_feature_prob=0.0,225        mask_feature_length=10,226        mask_feature_min_masks=0,227        pad_token_id=1,228        bos_token_id=0,229        eos_token_id=2,230        decoder_start_token_id=2,231        num_mel_bins=80,232        speech_decoder_prenet_layers=2,233        speech_decoder_prenet_units=256,234        speech_decoder_prenet_dropout=0.5,235        speaker_embedding_dim=512,236        speech_decoder_postnet_layers=5,237        speech_decoder_postnet_units=256,238        speech_decoder_postnet_kernel=5,239        speech_decoder_postnet_dropout=0.5,240        reduction_factor=2,241        max_speech_positions=4000,242        max_text_positions=450,243        encoder_max_relative_position=160,244        use_guided_attention_loss=True,245        guided_attention_loss_num_heads=2,246        guided_attention_loss_sigma=0.4,247        guided_attention_loss_scale=10.0,248        use_cache=True,249        is_encoder_decoder=True,250        **kwargs,251    ):252        self.vocab_size = vocab_size253        self.hidden_size = hidden_size254        self.encoder_layers = encoder_layers255        self.encoder_ffn_dim = encoder_ffn_dim256        self.encoder_attention_heads = encoder_attention_heads257        self.encoder_layerdrop = encoder_layerdrop258        self.decoder_layers = decoder_layers259        self.decoder_ffn_dim = decoder_ffn_dim260        self.decoder_attention_heads = decoder_attention_heads261        self.decoder_layerdrop = decoder_layerdrop262        self.hidden_act = hidden_act263        self.positional_dropout = positional_dropout264        self.hidden_dropout = hidden_dropout265        self.attention_dropout = attention_dropout266        self.activation_dropout = activation_dropout267        self.initializer_range = initializer_range268        self.layer_norm_eps = layer_norm_eps269        self.scale_embedding = scale_embedding270 271        self.feat_extract_norm = feat_extract_norm272        self.feat_proj_dropout = feat_proj_dropout273        self.feat_extract_activation = feat_extract_activation274        self.conv_dim = list(conv_dim)275        self.conv_stride = list(conv_stride)276        self.conv_kernel = list(conv_kernel)277        self.conv_bias = conv_bias278        self.num_conv_pos_embeddings = num_conv_pos_embeddings279        self.num_conv_pos_embedding_groups = num_conv_pos_embedding_groups280        self.num_feat_extract_layers = len(self.conv_dim)281 282        if (283            (len(self.conv_stride) != self.num_feat_extract_layers)284            or (len(self.conv_kernel) != self.num_feat_extract_layers)285            or (len(self.conv_dim) != self.num_feat_extract_layers)286        ):287            raise ValueError(288                "Configuration for convolutional layers is incorrect. It is required that `len(config.conv_dim)` =="289                " `len(config.conv_stride)` == `len(config.conv_kernel)`, but is `len(config.conv_dim) ="290                f" {len(self.conv_dim)}`, `len(config.conv_stride) = {len(self.conv_stride)}`,"291                f" `len(config.conv_kernel) = {len(self.conv_kernel)}`."292            )293 294        # fine-tuning config parameters for SpecAugment: https://huggingface.co/papers/1904.08779295        self.apply_spec_augment = apply_spec_augment296        self.mask_time_prob = mask_time_prob297        self.mask_time_length = mask_time_length298        self.mask_time_min_masks = mask_time_min_masks299        self.mask_feature_prob = mask_feature_prob300        self.mask_feature_length = mask_feature_length301        self.mask_feature_min_masks = mask_feature_min_masks302 303        self.num_mel_bins = num_mel_bins304        self.speech_decoder_prenet_layers = speech_decoder_prenet_layers305        self.speech_decoder_prenet_units = speech_decoder_prenet_units306        self.speech_decoder_prenet_dropout = speech_decoder_prenet_dropout307        self.speaker_embedding_dim = speaker_embedding_dim308 309        self.speech_decoder_postnet_layers = speech_decoder_postnet_layers310        self.speech_decoder_postnet_units = speech_decoder_postnet_units311        self.speech_decoder_postnet_kernel = speech_decoder_postnet_kernel312        self.speech_decoder_postnet_dropout = speech_decoder_postnet_dropout313        self.reduction_factor = reduction_factor314 315        self.max_speech_positions = max_speech_positions316        self.max_text_positions = max_text_positions317        self.encoder_max_relative_position = encoder_max_relative_position318 319        self.use_guided_attention_loss = use_guided_attention_loss320        self.guided_attention_loss_num_heads = guided_attention_loss_num_heads321        self.guided_attention_loss_sigma = guided_attention_loss_sigma322        self.guided_attention_loss_scale = guided_attention_loss_scale323 324        self.use_cache = use_cache325        self.is_encoder_decoder = is_encoder_decoder326 327        super().__init__(328            pad_token_id=pad_token_id,329            bos_token_id=bos_token_id,330            eos_token_id=eos_token_id,331            is_encoder_decoder=is_encoder_decoder,332            decoder_start_token_id=decoder_start_token_id,333            **kwargs,334        )335 336    def inputs_to_logits_ratio(self):337        return functools.reduce(operator.mul, self.conv_stride, 1)338 339 340class SpeechT5HifiGanConfig(PretrainedConfig):341    r"""342    This is the configuration class to store the configuration of a [`SpeechT5HifiGanModel`]. It is used to instantiate343    a SpeechT5 HiFi-GAN vocoder model according to the specified arguments, defining the model architecture.344    Instantiating a configuration with the defaults will yield a similar configuration to that of the SpeechT5345    [microsoft/speecht5_hifigan](https://huggingface.co/microsoft/speecht5_hifigan) architecture.346 347    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the348    documentation from [`PretrainedConfig`] for more information.349 350    Args:351        model_in_dim (`int`, *optional*, defaults to 80):352            The number of frequency bins in the input log-mel spectrogram.353        sampling_rate (`int`, *optional*, defaults to 16000):354            The sampling rate at which the output audio will be generated, expressed in hertz (Hz).355        upsample_initial_channel (`int`, *optional*, defaults to 512):356            The number of input channels into the upsampling network.357        upsample_rates (`tuple[int]` or `list[int]`, *optional*, defaults to `[4, 4, 4, 4]`):358            A tuple of integers defining the stride of each 1D convolutional layer in the upsampling network. The359            length of *upsample_rates* defines the number of convolutional layers and has to match the length of360            *upsample_kernel_sizes*.361        upsample_kernel_sizes (`tuple[int]` or `list[int]`, *optional*, defaults to `[8, 8, 8, 8]`):362            A tuple of integers defining the kernel size of each 1D convolutional layer in the upsampling network. The363            length of *upsample_kernel_sizes* defines the number of convolutional layers and has to match the length of364            *upsample_rates*.365        resblock_kernel_sizes (`tuple[int]` or `list[int]`, *optional*, defaults to `[3, 7, 11]`):366            A tuple of integers defining the kernel sizes of the 1D convolutional layers in the multi-receptive field367            fusion (MRF) module.368        resblock_dilation_sizes (`tuple[tuple[int]]` or `list[list[int]]`, *optional*, defaults to `[[1, 3, 5], [1, 3, 5], [1, 3, 5]]`):369            A nested tuple of integers defining the dilation rates of the dilated 1D convolutional layers in the370            multi-receptive field fusion (MRF) module.371        initializer_range (`float`, *optional*, defaults to 0.01):372            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.373        leaky_relu_slope (`float`, *optional*, defaults to 0.1):374            The angle of the negative slope used by the leaky ReLU activation.375        normalize_before (`bool`, *optional*, defaults to `True`):376            Whether or not to normalize the spectrogram before vocoding using the vocoder's learned mean and variance.377 378    Example:379 380    ```python381    >>> from transformers import SpeechT5HifiGan, SpeechT5HifiGanConfig382 383    >>> # Initializing a "microsoft/speecht5_hifigan" style configuration384    >>> configuration = SpeechT5HifiGanConfig()385 386    >>> # Initializing a model (with random weights) from the "microsoft/speecht5_hifigan" style configuration387    >>> model = SpeechT5HifiGan(configuration)388 389    >>> # Accessing the model configuration390    >>> configuration = model.config391    ```"""392 393    model_type = "hifigan"394 395    def __init__(396        self,397        model_in_dim=80,398        sampling_rate=16000,399        upsample_initial_channel=512,400        upsample_rates=[4, 4, 4, 4],401        upsample_kernel_sizes=[8, 8, 8, 8],402        resblock_kernel_sizes=[3, 7, 11],403        resblock_dilation_sizes=[[1, 3, 5], [1, 3, 5], [1, 3, 5]],404        initializer_range=0.01,405        leaky_relu_slope=0.1,406        normalize_before=True,407        **kwargs,408    ):409        self.model_in_dim = model_in_dim410        self.sampling_rate = sampling_rate411        self.upsample_initial_channel = upsample_initial_channel412        self.upsample_rates = upsample_rates413        self.upsample_kernel_sizes = upsample_kernel_sizes414        self.resblock_kernel_sizes = resblock_kernel_sizes415        self.resblock_dilation_sizes = resblock_dilation_sizes416        self.initializer_range = initializer_range417        self.leaky_relu_slope = leaky_relu_slope418        self.normalize_before = normalize_before419        super().__init__(**kwargs)420 421 422__all__ = ["SpeechT5Config", "SpeechT5HifiGanConfig"]423