CoolFace
Modelpublic

OPEA/MiniMax-Text-01-int4-sym-inc-preview

sourceHugging Faceupdated 1y agoView on Hugging Face
1likes8downloads
Model Card

Model Details

This model is an int4 model with group_size 128 and symmetric quantization of MiniMaxAI/MiniMax-Text-01 generated by intel/auto-round algorithm. This model is in AutoRound format, which is NOT supported by other serving frameworks, such as vLLM.

Please follow the license of the original model.

INT4 Inference on CUDA(480G*)

Requirements

bash
pip3 install git+https://github.com/intel/auto-round.git
pip3 install auto-gptq

This model is prone to overflow when running with int4 kernel with FP16 computation dtype and does not support CPU, as it explicitly relies on CUDA operations in the model files. While we have implemented several workarounds to ensure functionality, some prompts may still produce unexpected and random outputs.

~~~python from auto_round import AutoRoundConfig ##must import for autoround format from transformers import AutoModelForCausalLM, AutoTokenizer import torch

quantizedmodeldir = "OPEA/MiniMax-Text-01-int4-sym-inc-preview"

tokenizer = AutoTokenizer.frompretrained(quantizedmodeldir, trustremotecode=True) model = AutoModelForCausalLM.frompretrained(quantizedmodeldir, trustremotecode=True, torchdtype=torch.bfloat16, devicemap="auto")

def forward_hook(module, input, output): return torch.clamp(output, -65504, 65504).to(torch.bfloat16)

def registerfp16hooks(model): for name, module in model.namedmodules(): if "QuantLinear" in module.class.name or isinstance(module, torch.nn.Linear): module.registerforwardhook(forwardhook)

registerfp16hooks(model) tokenizer.padtoken = tokenizer.eostoken

prompts = [ "为什么企鹅没有被北极熊吃掉？", "树枝上有十只鸟，如果你射杀了一只，还剩下几只？请用中文回答", "How many r in strawberry.", "There is a girl who likes adventure,", "hello" ]

texts = [] for prompt in prompts: messages = [ {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant created by MiniMax based on MiniMax-Text-01 model."}]}, {"role": "user", "content": [{"type": "text", "text": prompt}]}, ] text = tokenizer.applychattemplate( messages, tokenize=False, addgenerationprompt=True ) texts.append(text) inputs = tokenizer(texts, returntensors="pt", padding=True, truncation=True, paddingside='left')

outputs = model.generate( inputids=inputs["inputids"].to(model.device), attentionmask=inputs["attentionmask"].to(model.device), maxnewtokens=512, numreturnsequences=1, dosample=False, eostokenid=200020, ) generatedids = [ outputids[len(inputids):] for inputids, outputids in zip(inputs["input_ids"], outputs) ]

decodedoutputs = tokenizer.batchdecode(generatedids, skipspecial_tokens=True)

for i, prompt in enumerate(prompts): inputid = inputs print(f"Prompt: {prompt}") print(f"Generated: {decodedoutputs[i]}") print("-" * 50)

""" Prompt: 为什么企鹅没有被北极熊吃掉？ Generated: ### 1. 地理分布差异

  • —企鹅：主要生活在南半球，例如南极洲。在南极洲，企鹅没有天敌，因为这里的环境非常恶劣，食物资源有限，动物数量也有限，企鹅是这里的顶级掠食者之一。
  • —北极熊：主要生活在北半球，例如北极地区。北极熊是北极地区的顶级掠食者之一，它们以海豹等动物为食。
  • —结论：由于地理分布的差异，企鹅和北极熊在自然界中无法相遇，因此北极熊无法吃掉企鹅。

2. 人为因素

  • —动物园或水族馆：在人为因素的影响 -------------------------------------------------- Prompt: 树枝上有十只鸟，如果你射杀了一只，还剩下几只？请用中文回答 Generated: 让我一步步思考这个问题：
  1. 1.首先,树枝上有10只鸟
  2. 2.射杀1只后,还剩9只
  3. 3.但实际上,当枪声响起,其他鸟会因惊吓而飞走
  4. 4.所以,当射杀1只后,树上不会剩下任何鸟

因此,答案是:0只

因为鸟会因枪声而飞走,不会继续停留在树上。 -------------------------------------------------- Prompt: How many r in strawberry. Generated: Let me solve this step by step.

  1. 1.First, let me count the r's in "strawberry" as I say it
  2. 2.s (not r)
  3. 3.t (not r)
  4. 4.r (1st r)
  5. 5.a (not r)
  6. 6.w (not r)
  7. 7.b (not r)
  8. 8.b (not r)
  9. 9.e (not r)
  10. 10.r (2nd r)
  11. 11.r (3rd r)
  12. 12.y (not r)
  1. 1.Counting the r's: 3 r's

Therefore, there is 3 r in strawberry.

The answer is 3. -------------------------------------------------- Prompt: There is a girl who likes adventure, Generated: There is a girl who likes adventure, and her name is Emily. Emily has always been drawn to the thrill of the unknown, the excitement of stepping into uncharted territory. Here is a story about -------------------------------------------------- Prompt: hello Generated: Hello! How can I assist you today? -------------------------------------------------- """ ~~~

Generated the model (2*80G)

pip3 install git+https://github.com/intel/auto-round.git

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "MiniMaxAI/MiniMax-Text-01"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, torch_dtype=torch.bfloat16)

fp_layers = [f"model.layers.{i}.block_sparse_moe.gate" for i in range(model.config.num_hidden_layers)]
layer_config = {}
for fp_layer in fp_layers:
    layer_config[fp_layer] = {"bits": 16}
    
device_map = {}
for i in range(32):
    key = fr"model\.layers\.\d+\.block_sparse_moe\.experts\.{str(i)}\..*$"
    if i < 14:
        device_map[key] = 0
    else:
        device_map[key] = 1


from auto_round import AutoRound

autoround = AutoRound(model=model, tokenizer=tokenizer, layer_config=layer_config, device_map=device_map,
                       batch_size=1,gradient_accumulate_steps=4, seqlen=512)
autoround.quantize()
autoround.save_quantized(format="auto_round", output_dir="tmp_autoround")

Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Therefore, before deploying any applications of the model, developers should perform safety testing.

Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

Here are a couple of useful links to learn more about Intel's AI software:

  • —Intel Neural Compressor link

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

Cite

@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }

arxiv github