Modelpublic

amd/Kimi-K3-Quark-MXFP4-AttnFP8

sourceHugging Faceotherupdated 28d agoView on Hugging Face

0likes797downloads

Model Card

Model Overview

—Model Architecture: KimiK3ForConditionalGeneration
—Input: Text, Image, Video
—Output: Text
—Supported Hardware Microarchitecture: AMD MI350/MI355
—ROCm: 7.1.1
—PyTorch: 2.10.0
—Transformers: 5.2.0
—Operating System(s): Linux
—Inference Engine: vLLM
—Model Optimizer: AMD-Quark
—experts, shared_experts: OCP MXFP4
—self_attn: PTPC-FP8

Model Quantization

The model was quantized from moonshotai/Kimi-K3 using AMD-Quark. Weights and activations are quantized to OCP MXFP4.

Quantization scripts:

python

from quark.torch import LLMTemplate, ModelQuantizer

# --- Register template ---
kimi_k3_template = LLMTemplate(
    model_type="kimi_k3",
    kv_layers_name=[
        "*self_attn.k_proj",
        "*self_attn.v_proj",
        "*self_attn.kv_a_proj_with_mqa",
    ],
    q_layer_name=[
        "*self_attn.q_proj",
        "*self_attn.q_a_proj",
        "*self_attn.q_b_proj",
    ],
    exclude_layers_name=[
        "*block_sparse_moe.gate*",
        "*block_sparse_moe.routed_expert_down_proj*",
        "*block_sparse_moe.routed_expert_up_proj*",
        "*lm_head*",
        "*vision_tower*",
        "*mm_projector*",
        "*self_attention_res_proj*",
        "*mlp_res_proj*",
        "*output_attn_res_proj*",
        "*self_attn.*_conv1d*",
    ],
)
LLMTemplate.register_template(kimi_k3_template)
print(f"[INFO]: Registered template '{kimi_k3_template.model_type}'")

# --- Configuration ---
model_dir = "moonshotai/Kimi-K3"
output_dir = "amd/Kimi-K3-Quark-MXFP4"
quant_scheme = "mxfp4"

#   --layer_quant_scheme "*self_attn*" ptpc_fp8
layer_config = {
    "*self_attn*": "ptpc_fp8",
}

# --- Build quant config from template ---
template = LLMTemplate.get("kimi_k3")
quant_config = template.get_config(
    scheme=quant_scheme,
    layer_config=layer_config,
)

# --- File-to-file quantization ---
quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=model_dir,
    save_path=output_dir,
)
print(f"[INFO]: Quantization complete. Output saved to {output_dir}")

Evaluation

The model can be evaluated using the vllm or SGLang framework.

Accuracy

Benchmark	moonshotai/Kimi-K3	amd/Kimi-K3-Quark-MXFP4 (this model)	Recovery
gsm8k (flexible-extract, 5-shot)	96.82	96.36	99.52%

Reproduction

The GSM8K results were obtained using the vLLM framework, based on the Docker image vllm/vllm-openai-rocm:kimi-k3, and vLLM is installed inside the container.

In this image, edit: /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/quark/schemes/quark_ocp_mx.py

In QuarkOCP_MX.__init__, around lines 209-212, replace:

python

self.emulate = not current_platform.supports_mx() or (
    self.input_dtype != "mxfp4" or self.weight_dtype != "mxfp4"
)

with:

python

self.emulate = True

Launching server

bash

VLLM_ROCM_USE_AITER=1 \
SAFETENSORS_FAST_GPU=1 \
VLLM_USE_BREAKABLE_CUDAGRAPH=0 \
vllm serve amd/Kimi-K3-Quark-MXFP4 \
  --trust-remote-code \
  --moe-backend auto \
  --tensor-parallel-size 8 \
  --load-format auto \
  --gpu-memory-utilization 0.95 \
  --mm-encoder-tp-mode data \
  --max-num-seqs 128 \
  --max-num-batched-tokens 4096 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

Evaluating model in a new terminal

bash

lm_eval \
  --model local-chat-completions \
  --model_args "model=amd/Kimi-K3-Quark-MXFP4,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=32,max_gen_toks=2048" \
  --tasks gsm8k \
  --num_fewshot 5 \
  --batch_size 1 \
  --apply_chat_template \
  --fewshot_as_multiturn

Model Overview

Model Quantization

Evaluation

Accuracy

Reproduction

Launching server

Evaluating model in a new terminal

License