Modelpublic

amd/GLM-5.1-MXFP4

sourceHugging Facemitupdated 3mo agoView on Hugging Face

1likes2.6kdownloads

Model Card

Model Overview

—Model Architecture: GLM-5.1
—Input: Text
—Output: Text
—Supported Hardware Microarchitecture: AMD MI350/MI355
—ROCm: 7.0.0
—PyTorch: 2.10.0
—Transformers: 5.2.0
—Operating System(s): Linux
—Inference Engine: vLLM
—Model Optimizer: AMD-Quark
—Weight quantization: MOE-only (shared experts quantized), OCP MXFP4, Static
—Activation quantization: MOE-only, OCP MXFP4, Dynamic
—Calibration Dataset: Pile

This model was built with GLM-5.1 model by applying AMD-Quark for MXFP4 quantization.

Model Quantization

The model was quantized from zai-org/GLM-5.1 using AMD-Quark. The weights and activations are quantized to MXFP4.

Quantization scripts:

python

from quark.torch import LLMTemplate, ModelQuantizer
# --- Register template ---
GLM5_template = LLMTemplate(
    model_type="glm_moe_dsa",
    kv_layers_name=["*kv_a_proj_with_mqa", "*kv_b_proj"],
    q_layer_name="*q_a_proj",
    exclude_layers_name=["lm_head"],
)
LLMTemplate.register_template(GLM5_template)
print(f"[INFO]: Registered template '{GLM5_template.model_type}'")
# --- Configuration ---
model_dir = "zai-org/GLM-5.1"
output_dir = "amd/GLM-5.1-MXFP4"
quant_scheme = "mxfp4"
exclude_layers = [
    "*self_attn*",
    "*mlp.gate",
    "*lm_head",
    "*mlp.gate_proj",
    "*mlp.up_proj",
    "*mlp.down_proj",
]
# --- Build quant config from template ---
template = LLMTemplate.get("glm_moe_dsa")
quant_config = template.get_config(scheme=quant_scheme, exclude_layers=exclude_layers)
# --- File-to-file quantization (memory-efficient, no full model loading) ---
quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=model_dir,
    save_path=output_dir,
)
print(f"[INFO]: Quantization complete. Output saved to {output_dir}")

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend.

Evaluation

The model was evaluated on GSM8K benchmarks.

Accuracy

<table> <tr> <td><strong>Benchmark</strong> </td> <td><strong>GLM-5.1 </strong> </td> <td><strong>GLM-5.1-MXFP4(this model)</strong> </td> <td><strong>Recovery</strong> </td> </tr> <tr> <td>GSM8K (flexible-extract) </td> <td>95.22 </td> <td>94.54 </td> <td>99.3% </td> </tr> </table>

Reproduction

The GSM8K results were obtained using the lm-evaluation-harness framework, based on the Docker image rocm/vllm-dev:nightly_main_20260526, with vLLM pre-installed inside the image and lm-eval compiled and installed from source.

Launching server

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FP8BMM=0
export VLLM_ROCM_USE_AITER_FP4BMM=0
vllm serve amd/GLM-5.1-MXFP4 \
  -tp 8 \
  --block-size 1 \
  --trust-remote-code \
  --max-model-len 4096

Evaluating model in a new terminal

lm_eval \
  --model local-completions \
  --model_args '{"model": "amd/GLM-5.1-MXFP4", "base_url": "http://localhost:8000/v1/completions", "num_concurrent": 32, "max_retries": 10, "max_gen_toks": 2048, "tokenizer_backend":"None","tokenized_requests":"False" }' \
  --tasks gsm8k \
  --batch_size auto \
  --num_fewshot 5 \
  --trust_remote_code