Modelpublic

amd/DeepSeek-V4.1-Flash-Quark-MXFP4

sourceHugging Facemitupdated 2d agoView on Hugging Face

0likes12downloads

Model Card

DeepSeek-V4.1-Flash-MXFP4

Model Overview

—Model Architecture: DeepseekV41ForCausalLM
—Input: Text, Image
—Output: Text
—Supported Hardware Microarchitecture: AMD MI355 / MI350 (gfx950)
—ROCm: 7.2.0
—PyTorch: 2.12.0
—Transformers: 5.17.0
—Operating System(s): Linux
—Inference Engine: vLLM
—Model Optimizer: AMD-Quark (v0.13.0)
—Quantized layers: experts, shared_experts, and self_attn in language model. The vision part is not quantized.
—experts and shared_experts: OCP MXFP4 for both weights and activations
—self_attn: OCP MXFP8 for both weights and activations

Model Quantization

Quantized from deepseek-ai/DeepSeek-V4.1-Flash with AMD Quark. The pipeline re-quantizes only the MoE expert weights and activations to MXFP4. All non-expert modules are kept as-is via the exclude list.

Quantization script

python

from quark.torch import LLMTemplate, ModelQuantizer

MODEL_DIR = "deepseek-ai/DeepSeek-V4.1-Flash"
OUTPUT_DIR = "amd/DeepSeek-V4.1-Flash-MXFP4"

template = LLMTemplate(
    model_type="deepseek_v41",
    kv_layers_name=["*wkv"],
    q_layer_name=["*wq_a", "*wq_b"],
    exclude_layers_name=[
        "*attn*",
        "embed",
        "*head*",
        "*ffn.gate*",
        "hc_*",
        "*engram*",
        "*vision*",
        "*aligner*",
        "*main_proj*",
    ],
)

LLMTemplate.register_template(template)

quant_config = template.get_config(scheme="mxfp4")

quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=MODEL_DIR,
    save_path=OUTPUT_DIR,
    keep_excluded_layers_as_original_model_state=True,
)

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend based on the Docker image vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657.

Evaluation

The model was evaluated on GSM8K (8-shot) and GPQA Diamond benchmarks using the vLLM framework.

Accuracy

Benchmark	deepseek-ai/DeepSeek-V4.1-Flash	amd/DeepSeek-V4.1-Flash-MXFP4 (this model)	Recovery
GSM8K (flexible-extract)	92.87	92.34	99.4%
GPQA Diamond (flexible-extract)	90.40	87.88	97.2%

Reproduction

The evaluation results were obtained using the lm-eval framework with the vLLM direct integration backend, based on the Docker image vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657.

Launching server

bash

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MHA=1
export VLLM_ROCM_USE_AITER_MOE=0
vllm serve amd/DeepSeek-V4.1-Flash-MXFP4 --tensor-parallel-size 4 \
  --trust-remote-code --tokenizer-mode deepseek_v41 \
  --max-model-len 8192 --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90 --enforce-eager

Evaluating GSM8K in a new terminal

bash

lm_eval --model local-completions \
    --model_args model=amd/DeepSeek-V4.1-Flash-MXFP4,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,num_concurrent=8,tokenizer=amd/DeepSeek-V4.1-Flash-MXFP4 \
    --tasks gsm8k --batch_size auto --num_fewshot 8 --seed 42 \
    --gen_kwargs "temperature=0,max_gen_toks=4096"

Evaluating GPQA Diamond

bash

lm_eval --model vllm \
    --model_args '{"pretrained":"amd/DeepSeek-V4.1-Flash-MXFP4","tokenizer":"amd/DeepSeek-V4.1-Flash-MXFP4","tensor_parallel_size":4,"dtype":"auto","gpu_memory_utilization":0.9,"enforce_eager":true,"trust_remote_code":true,"max_length":73728,"enable_thinking":true,"think_end_token":"</think>","chat_template_args":{"reasoning_effort":100},"max_num_batched_tokens":8192,"tokenizer_mode":"deepseek_v41"}' \
    --tasks gpqa_diamond_generative_n_shot \
    --apply_chat_template --fewshot_as_multiturn \
    --gen_kwargs "max_gen_toks=65536,do_sample=True,temperature=1.0,top_p=0.95,top_k=0,min_p=0.0,presence_penalty=0.0,repetition_penalty=1.0" \
    --batch_size auto --log_samples

License

This model is a quantized derivative of deepseek-ai/DeepSeek-V4.1-Flash and is distributed under the same license as the source model: the MIT License. A copy of the upstream LICENSE is included in this repository.

Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. AMD has modified the model weights of the MoE expert layers by quantizing them to MXFP4 with AMD Quark; the modifications are provided under the same MIT License and are not subject to any separate or different license.