MiniMaxAI/MiniMax-M2.7

1#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨2#           This file was automatically generated from src/transformers/models/minimax_m2/modular_minimax_m2.py.3#               Do NOT edit this file manually as any edits will be overwritten by the generation of4#             the file from the modular. If any change should be done, please apply the change to the5#                          modular_minimax_m2.py file directly. One of our CI enforces this.6#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨7# coding=utf-88# Copyright 2025 the HuggingFace Team. All rights reserved.9#10# Licensed under the Apache License, Version 2.0 (the "License");11# you may not use this file except in compliance with the License.12# You may obtain a copy of the License at13#14#     http://www.apache.org/licenses/LICENSE-2.015#16# Unless required by applicable law or agreed to in writing, software17# distributed under the License is distributed on an "AS IS" BASIS,18# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.19# See the License for the specific language governing permissions and20# limitations under the License.21 22 23from transformers.configuration_utils import PretrainedConfig24 25 26class MiniMaxM2Config(PretrainedConfig):27    r"""28    This is the configuration class to store the configuration of a [`MiniMaxM2Model`]. It is used to instantiate an29    MiniMaxM2 model according to the specified arguments, defining the model architecture. Instantiating a configuration30    with the defaults will yield a similar configuration to that of the MiniMaxM2-7B-v0.1 or MiniMaxM2-7B-Instruct-v0.1.31 32    [minimax_m2ai/MiniMaxM2-8x7B](https://huggingface.co/minimax_m2ai/MiniMaxM2-8x7B)33    [minimax_m2ai/MiniMaxM2-7B-Instruct-v0.1](https://huggingface.co/minimax_m2ai/MiniMaxM2-7B-Instruct-v0.1)34 35    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the36    documentation from [`PretrainedConfig`] for more information.37 38 39    Args:40        vocab_size (`int`, *optional*, defaults to 32000):41            Vocabulary size of the MiniMaxM2 model. Defines the number of different tokens that can be represented by the42            `inputs_ids` passed when calling [`MiniMaxM2Model`]43        hidden_size (`int`, *optional*, defaults to 4096):44            Dimension of the hidden representations.45        intermediate_size (`int`, *optional*, defaults to 14336):46            Dimension of the MLP representations.47        num_hidden_layers (`int`, *optional*, defaults to 32):48            Number of hidden layers in the Transformer encoder.49        num_attention_heads (`int`, *optional*, defaults to 32):50            Number of attention heads for each attention layer in the Transformer encoder.51        num_key_value_heads (`int`, *optional*, defaults to 8):52            This is the number of key_value heads that should be used to implement Grouped Query Attention. If53            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if54            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When55            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed56            by meanpooling all the original heads within that group. For more details, check out [this57            paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `8`.58        head_dim (`int`, *optional*, defaults to `hidden_size // num_attention_heads`):59            The attention head dimension.60        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):61            The non-linear activation function (function or string) in the decoder.62        max_position_embeddings (`int`, *optional*, defaults to `4096*32`):63            The maximum sequence length that this model might ever be used with. MiniMaxM2's sliding window attention64            allows sequence of up to 4096*32 tokens.65        initializer_range (`float`, *optional*, defaults to 0.02):66            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.67        rms_norm_eps (`float`, *optional*, defaults to 1e-05):68            The epsilon used by the rms normalization layers.69        use_cache (`bool`, *optional*, defaults to `True`):70            Whether or not the model should return the last key/values attentions (not used by all models). Only71            relevant if `config.is_decoder=True`.72        pad_token_id (`int`, *optional*):73            The id of the padding token.74        bos_token_id (`int`, *optional*, defaults to 1):75            The id of the "beginning-of-sequence" token.76        eos_token_id (`int`, *optional*, defaults to 2):77            The id of the "end-of-sequence" token.78        tie_word_embeddings (`bool`, *optional*, defaults to `False`):79            Whether the model's input and output word embeddings should be tied.80        rope_theta (`float`, *optional*, defaults to 1000000.0):81            The base period of the RoPE embeddings.82        sliding_window (`int`, *optional*):83            Sliding window attention window size. If not specified, will default to `4096`.84        attention_dropout (`float`, *optional*, defaults to 0.0):85            The dropout ratio for the attention probabilities.86        num_experts_per_tok (`int`, *optional*, defaults to 2):87            The number of experts to route per-token, can be also interpreted as the `top-k` routing88            parameter89        num_local_experts (`int`, *optional*, defaults to 8):90            Number of experts per Sparse MLP layer.91        output_router_logits (`bool`, *optional*, defaults to `False`):92            Whether or not the router logits should be returned by the model. Enabling this will also93            allow the model to output the auxiliary loss. See [here]() for more details94        router_aux_loss_coef (`float`, *optional*, defaults to 0.001):95            The aux loss factor for the total loss.96        router_jitter_noise (`float`, *optional*, defaults to 0.0):97            Amount of noise to add to the router.98 99    ```python100    >>> from transformers import MiniMaxM2Model, MiniMaxM2Config101 102    >>> # Initializing a MiniMaxM2 7B style configuration103    >>> configuration = MiniMaxM2Config()104 105    >>> # Initializing a model from the MiniMaxM2 7B style configuration106    >>> model = MiniMaxM2Model(configuration)107 108    >>> # Accessing the model configuration109    >>> configuration = model.config110    ```"""111 112    model_type = "minimax_m2"113    keys_to_ignore_at_inference = ["past_key_values"]114    base_model_tp_plan = {115        "layers.*.self_attn.q_proj": "colwise",116        "layers.*.self_attn.k_proj": "colwise",117        "layers.*.self_attn.v_proj": "colwise",118        "layers.*.self_attn.o_proj": "rowwise",119        "layers.*.block_sparse_moe.gate": "colwise_rep",  # we need to replicate here to correctly route experts120        "layers.*.block_sparse_moe.experts.*.w1": "colwise",121        "layers.*.block_sparse_moe.experts.*.w2": "rowwise",122        "layers.*.block_sparse_moe.experts.*.w3": "colwise",123    }124    base_model_pp_plan = {125        "embed_tokens": (["input_ids"], ["inputs_embeds"]),126        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),127        "norm": (["hidden_states"], ["hidden_states"]),128    }129 130    def __init__(131        self,132        vocab_size=32000,133        hidden_size=4096,134        intermediate_size=14336,135        num_hidden_layers=32,136        num_attention_heads=32,137        num_key_value_heads=8,138        head_dim=None,139        hidden_act="silu",140        max_position_embeddings=4096 * 32,141        initializer_range=0.02,142        rms_norm_eps=1e-5,143        use_cache=True,144        pad_token_id=None,145        bos_token_id=1,146        eos_token_id=2,147        tie_word_embeddings=False,148        rope_theta=1e6,149        sliding_window=None,150        attention_dropout=0.0,151        num_experts_per_tok=2,152        num_local_experts=8,153        output_router_logits=False,154        router_aux_loss_coef=0.001,155        router_jitter_noise=0.0,156        **kwargs,157    ):158        self.vocab_size = vocab_size159        self.max_position_embeddings = max_position_embeddings160        self.hidden_size = hidden_size161        self.intermediate_size = intermediate_size162        self.num_hidden_layers = num_hidden_layers163        self.num_attention_heads = num_attention_heads164        self.sliding_window = sliding_window165 166        # for backward compatibility167        if num_key_value_heads is None:168            num_key_value_heads = num_attention_heads169 170        self.num_key_value_heads = num_key_value_heads171        self.hidden_act = hidden_act172        self.initializer_range = initializer_range173        self.rms_norm_eps = rms_norm_eps174        self.use_cache = use_cache175        self.rope_theta = rope_theta176        self.attention_dropout = attention_dropout177        self.head_dim = head_dim178 179        self.num_experts_per_tok = num_experts_per_tok180        self.num_local_experts = num_local_experts181        self.output_router_logits = output_router_logits182        self.router_aux_loss_coef = router_aux_loss_coef183        self.router_jitter_noise = router_jitter_noise184 185        self.use_qk_norm = kwargs.pop("use_qk_norm", False)186        self.rotary_dim = kwargs.pop("rotary_dim", self.head_dim)187        self.partial_rotary_factor = kwargs.pop("partial_rotary_factor", 1)188        if self.head_dim is not None:189            self.partial_rotary_factor = self.rotary_dim / self.head_dim190 191        super().__init__(192            pad_token_id=pad_token_id,193            bos_token_id=bos_token_id,194            eos_token_id=eos_token_id,195            tie_word_embeddings=tie_word_embeddings,196            **kwargs,197        )198 199 200__all__ = ["MiniMaxM2Config"]201