amd/DeepSeek-R1-0528-MXFP4

1---2license: mit3base_model:4- deepseek-ai/DeepSeek-R1-05285---6 7 8# Model Overview9 10- **Model Architecture:** DeepSeek-R1-052811  - **Input:** Text12  - **Output:** Text13- **Supported Hardware Microarchitecture:** AMD MI350/MI35514- **ROCm**: 7.015- **PyTorch**: 2.8.016- **Transformers**: 4.53.017- **Operating System(s):** Linux18- **Inference Engine:** [SGLang](https://docs.sglang.ai/)/[vLLM](https://docs.vllm.ai/en/latest/)19- **Model Optimizer:** [AMD-Quark](https://quark.docs.amd.com/latest/index.html) (V0.10)20  - **Weight quantization:** OCP MXFP4, Static21  - **Activation quantization:** OCP MXFP4, Dynamic22- **Calibration Dataset:** [Pile](https://huggingface.co/datasets/mit-han-lab/pile-val-backup)23 24This model was built with deepseek-ai DeepSeek-R1-0528 model by applying [AMD-Quark](https://quark.docs.amd.com/latest/index.html) for MXFP4 quantization.25 26# Model Quantization27 28The model was quantized from [deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) using [AMD-Quark](https://quark.docs.amd.com/latest/index.html). Both weights and activations were quantized to MXFP4 format. 29 30**Preprocessing requirement:**31 32Before executing the quantization script below, the original FP8 model must first be dequantized to BFloat16.33You can either perform the dequantization manually using this [conversion script](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py), or use the pre-converted BFloat16 model available at [amd/DeepSeek-R1-0528-BF16](https://huggingface.co/amd/DeepSeek-R1-0528-BF16).34 35**Quantization scripts:**36```37cd Quark/examples/torch/language_modeling/llm_ptq/38exclude_layers="*self_attn* *mlp.gate.* *lm_head model.layers.61.*"39python3 quantize_quark.py --model_dir $MODEL_DIR \40                          --quant_scheme w_mxfp4_a_mxfp4 \41                          --group_size 32 \42                          --num_calib_data 128 \43                          --exclude_layers $exclude_layers \44                          --skip_evaluation \45                          --multi_gpu \46                          --model_export hf_format \47                          --output_dir amd/DeepSeek-R1-0528-MXFP448```49 50# Deployment51 52This model can be deployed efficiently using the [SGLang](https://docs.sglang.ai/) and [vLLM](https://docs.vllm.ai/en/latest/) backends.53 54## Evaluation55 56The model was evaluated on AIME24, GPQA Diamond, and MATH-500 benchmarks using the [lighteval](https://github.com/huggingface/lighteval/tree/v0.10.0) framework. Each benchmark was run 10 times with different random seeds for reliable performance estimation.57 58### Accuracy59 60<table>61  <tr>62   <td><strong>Benchmark</strong>63   </td>64   <td><strong>DeepSeek-R1-0528 </strong>65   </td>66   <td><strong>DeepSeek-R1-0528-MXFP4 (this model)</strong>67   </td>68   <td><strong>Recovery</strong>69   </td>70  </tr>71  <tr>72   <td>AIME24 73   </td>74   <td>88.0075   </td>76   <td>85.0077   </td>78   <td>96.59%79   </td>80  </tr>81  <tr>82   <td>GPQA Diamond 83   </td>84   <td>79.9085   </td>86   <td>79.3487   </td>88   <td>99.31%89   </td>90  </tr>91  <tr>92   <td>MATH-500 93   </td>94   <td>97.0695   </td>96   <td>97.8497   </td>98   <td>100.80%99   </td>100  </tr>101</table>102 103 104### Reproduction105 106The results of AIME24, MATH-500, and GPQA Diamond, were obtained using forked [lighteval](https://github.com/zhaolin-amd/lighteval/tree/v0.10-release-custom) and vLLM docker (emulation qdq) `rocm/vllm-private:pytorch-vllm-gfx950-mxfp4-mxfp6-v3`.107 108```109# Set docker env110export VLLM_QUARK_F4F6_OFFLINE_DEQUANT_TMPENVVAR=1111 112# Set model args113OUTPUT_DIR="results/DeepSeek-R1-0528-MXFP4-Seed"114LOG="logs/deepseek_0528_maxfp4.log"115 116# Evaluating 10 rounds 117for i in $(seq 1 10); do118    # seed in [0, 2**30 - 1]119    SEED=$(shuf -i 0-1073741823 -n 1)120    MODEL_ARGS="model_name=amd/DeepSeek-R1-0528-MXFP4,dtype=bfloat16,tensor_parallel_size=8,max_model_length=71536,max_num_batched_tokens=32768,gpu_memory_utilization=0.85,generation_parameters={max_new_tokens:65536,temperature:0.6,top_p:0.95,seed:$SEED}"121 122    lighteval vllm $MODEL_ARGS "custom|aime24_single|0|0,custom|math_500_single|0|0,custom|gpqa:diamond_single|0|0" \123        --use-chat-template \124        --output-dir "$OUTPUT_DIR/seed_$SEED" \125        2>&1 | tee -a "$LOG"126```127 128# License129Modifications Copyright(c) 2025 Advanced Micro Devices, Inc. All rights reserved.