amd/Qwen1.5-MoE-A2.7B-Chat-w-int8-a-int8-sym

Model Overview

• —Model Architecture: Qwen2MoeForCausalLM
• —Input: Text
• —Output: Text
• —Supported Hardware Microarchitecture: AMD MI300 / MI350 / MI355
• —Inference Engine: vLLM
• —Model Optimizer: AMD-Quark (v0.11.2)
• —Weight quantization: INT8, per-channel, Static, Symmetric
• —Activation quantization: INT8, per-token, Dynamic, Symmetric
• —Quantized layers: routed MoE experts only (attention, shared expert, and router/gate layers are kept in the original precision)

This is a W8A8-INT8 quantization of Qwen/Qwen1.5-MoE-A2.7B-Chat, used as vLLM CI test coverage for the Quark INT8 fused-MoE path (QuarkW8A8Int8MoEMethod).

Model Quantization

The model was quantized from Qwen/Qwen1.5-MoE-A2.7B-Chat using AMD-Quark. The routed expert weights are quantized to INT8 with per-channel static scales, and their activations are quantized to INT8 with per-token dynamic scales (symmetric).

Note: Quark quantizes nn.Linear modules. MoE experts are stored as individual nn.Linear layers in transformers ~4.57; quantize with that version so the routed experts are captured.

Quantization script (see `tests/quantization/quantize_int8_moe_quark.py` in vLLM):

python tests/quantization/quantize_int8_moe_quark.py \
    --model Qwen/Qwen1.5-MoE-A2.7B-Chat \
    --output-dir ./Qwen1.5-MoE-A2.7B-Chat-w-int8-a-int8-sym \
    --exclude lm_head '*.self_attn.*' '*.shared_expert.*' \
              '*.shared_expert_gate' '*.mlp.gate'

Evaluation

The model was evaluated on gsm8k using the vLLM framework (the routed experts run through vLLM's Triton INT8 fused-MoE kernel).

Accuracy

Reproduction

lm_eval \
  --model vllm \
  --model_args pretrained=amd/Qwen1.5-MoE-A2.7B-Chat-w-int8-a-int8-sym,max_model_len=4096,gpu_memory_utilization=0.4,enforce_eager=True,trust_remote_code=True \
  --tasks gsm8k --num_fewshot 5 \
  --batch_size auto

License

This model inherits the Tongyi Qianwen license of the base model.

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.