RedHatAI/DeepSeek-Coder-V2-Instruct-0724-quantized.w4a16

1---2language:3- en4tags:5- moe6- int47- w4a168- vllm9license: other10license_name: deepseek11license_link: https://github.com/deepseek-ai/DeepSeek-V2/blob/main/LICENSE-MODEL12library_name: transformers13---14 15# DeepSeek-Coder-V2-Instruct-0724-quantized.w4a1616 17## Model Overview18- **Model Architecture:** DeepSeek-Coder-V2-Instruct-072419  - **Input:** Text20  - **Output:** Text21- **Model Optimizations:**22  - **Weight quantization:** INT423  - **Activation quantization:** None24- **Release Date:** 3/1/202525- **Version:** 1.026- **Model Developers:** Neural Magic27 28Quantized version of [DeepSeek-Coder-V2-Instruct-0724](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct-0724).29 30### Model Optimizations31 32This model was obtained by quantizing only the weights to INT4 data type, ready for inference with vLLM >= 0.5.2.33This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. The weights of the linear operators within transformers blocks are quantized, except the MLP routers. 34 35## Deployment36 37### Use with vLLM38 39This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.40 41```python42from transformers import AutoTokenizer43from vllm import LLM, SamplingParams44 45max_model_len, tp_size = 4096, 246model_name = "neuralmagic-ent/DeepSeek-Coder-V2-Instruct-0724-quantized.w4a16"47tokenizer = AutoTokenizer.from_pretrained(model_name)48llm = LLM(model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True)49sampling_params = SamplingParams(temperature=0.3, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id])50 51messages_list = [52    [{"role": "user", "content": "Who are you? Please respond in pirate speak!"}],53]54 55prompt_token_ids = [tokenizer.apply_chat_template(messages, add_generation_prompt=True) for messages in messages_list]56 57outputs = llm.generate(prompt_token_ids=prompt_token_ids, sampling_params=sampling_params)58 59generated_text = [output.outputs[0].text for output in outputs]60print(generated_text)61```62 63vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.64 65## Creation66 67This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below with the following command:68 69```bash70python quantize.py --model_path deepseek-ai/DeepSeek-Coder-V2-Instruct-0724 --quant_path "output_dir" --calib_size 256 --dampening_frac 0.1 --observer mse --actorder False71```72 73 74```python75`from datasets import load_dataset76from transformers import AutoTokenizer77from llmcompressor.modifiers.quantization import GPTQModifier78from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot, apply79import argparse80from compressed_tensors.quantization import QuantizationScheme, QuantizationArgs, QuantizationType, QuantizationStrategy81from llmcompressor.transformers.compression.helpers import calculate_offload_device_map82import torch83 84 85def parse_actorder(value):86    # Interpret the input value for --actorder87    if value.lower() == "false":88        return False89    elif value.lower() == "weight":90        return "weight"91    elif value.lower() == "group":92        raise ValueError("group not supported for TP>1 and MoEs")93    else:94        raise argparse.ArgumentTypeError("Invalid value for --actorder. Use 'group' or 'False'.")95 96 97parser = argparse.ArgumentParser()98parser.add_argument('--model_path', type=str)99parser.add_argument('--quant_path', type=str)100parser.add_argument('--num_bits', type=int, default=4)101parser.add_argument('--sequential_update', type=bool, default=True)102parser.add_argument('--calib_size', type=int, default=256)103parser.add_argument('--dampening_frac', type=float, default=0.05)104parser.add_argument('--observer', type=str, default="minmax")105parser.add_argument(106    '--actorder',107    type=parse_actorder,108    default=False,  # Default value is False109    help="Specify actorder as 'group' (string) or False (boolean)."110)111 112args = parser.parse_args()113 114device_map = calculate_offload_device_map(115    args.model_path,116    reserve_for_hessians=True,117    num_gpus=torch.cuda.device_count(),118    torch_dtype=torch.bfloat16,119    trust_remote_code=True,120)121 122model = SparseAutoModelForCausalLM.from_pretrained(123    args.model_path,124    device_map=device_map,125    torch_dtype=torch.bfloat16,126    use_cache=False,127    trust_remote_code=True,128)129tokenizer = AutoTokenizer.from_pretrained(args.model_path)130 131NUM_CALIBRATION_SAMPLES = args.calib_size132DATASET_ID = "garage-bAInd/Open-Platypus"133DATASET_SPLIT = "train"134ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)135ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))136 137def preprocess(example):138    concat_txt = example["instruction"] + "\n" + example["output"]139    return {"text": concat_txt}140 141ds = ds.map(preprocess)142 143def tokenize(sample):144    return tokenizer(145        sample["text"],146        padding=False,147        truncation=False,148        add_special_tokens=True,149    )150 151 152ds = ds.map(tokenize, remove_columns=ds.column_names)153 154quant_scheme = QuantizationScheme(155    targets=["Linear"],156    weights=QuantizationArgs(157        num_bits=args.num_bits,158        type=QuantizationType.INT,159        symmetric=True,160        group_size=128,161        strategy=QuantizationStrategy.GROUP,162        observer=args.observer,163        actorder=args.actorder164    ),165    input_activations=None,166    output_activations=None,167)168 169recipe = [170    GPTQModifier(171        targets=["Linear"],172        ignore=["lm_head", "re:.*\.mlp\.gate$"],173        sequential_update=args.sequential_update,174        dampening_frac=args.dampening_frac,175        config_groups={"group_0": quant_scheme},176    )177]178oneshot(179    model=model,180    dataset=ds,181    recipe=recipe,182    num_calibration_samples=args.calib_size,183)184 185# Save to disk compressed.186SAVE_DIR = args.quant_path187model.save_pretrained(SAVE_DIR, save_compressed=True, skip_compression_stats=True)188tokenizer.save_pretrained(SAVE_DIR)189```190 191## Evaluation192 193The model was evaluated on [HumanEval and HumanEval+](https://github.com/openai/human-eval?tab=readme-ov-file) benchmark with the [Neural Magic fork](https://github.com/neuralmagic/evalplus) of the [EvalPlus implementation of HumanEval+](https://github.com/evalplus/evalplus) and the [vLLM](https://docs.vllm.ai/en/stable/) engine, using the following commands:194 195```196python evalplus/codegen/generate.py --model neuralmagic-ent/DeepSeek-Coder-V2-Instruct-0724-quantized.w4a16 --bs 16 --temperature 0.2 --n_samples 50 --root "./results" --dataset humaneval --backend vllm --dtype auto --tp 8 197 198python evalplus/evalplus/sanitize.py results/humaneval/neuralmagic-ent--DeepSeek-Coder-V2-Instruct-0724-quantized.w4a16_vllm_temp_0.2199 200evalplus.evaluate --dataset humaneval --samples results/humaneval/neuralmagic-ent--DeepSeek-Coder-V2-Instruct-0724-quantized.w4a16_vllm_temp_0.2-sanitized201```202 203 204### Accuracy205 206#### HumanEval evaluation scores207 208| Metric                 | deepseek-ai/DeepSeek-Coder-V2-Instruct-0724             | neuralmagic-ent/DeepSeek-Coder-V2-Instruct-0724-quantized.w4a16 |209|------------------------|:---------------------------------:|:-------------------------------------------:|210| HumanEval pass@1       | 89.3                            | 85.5                                        |211| HumanEval pass@10      | 93.1                            | 91.1                                        |212| HumanEval+ pass@1      | 82.9                            | 80.7                                       |213| HumanEval+ pass@10     | 87.6                            | 85.9                                       |214| **Average Score**                       | **88.23**                        | **85.8**                                   |215| **Recovery**                            | **100.00**                       | **97.25**                                   |216 217