amd/Instella-MoE-16B-A3B-SFT-w8a8-llmcompressor

Instella-MoE-16B-A3B-SFT-w8a8-llmcompressor

Model Overview

• —Model Architecture: InstellaMoEForCausalLM
• —Input: Text
• —Output: Text
• —Source Model: Instella-MoE-16B-A3B-SFT
• —Supported Hardware: AMD EPYC (CPU inference)
• —Preferred Operating System: Linux
• —Inference Engine: vLLM v0.28.0
• —Quantization Framework: LLM Compressor v0.13.0
• —Quantization Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
• —Compatible Stack:
• —ZenDNN v6.1.0
• —ZenTorch v2.13.0
• —PyTorch v2.13.0
• —LLM Compressor v0.13.0
• —vLLM v0.28.0
• —Published with: LLM Compressor v0.13.0

This is a quantized version of Instella-MoE-16B-A3B-SFT created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Instella-MoE-16B-A3B-SFT using LLM Compressor via the Round-to-Nearest (RTN) algorithm. This reduces the model weights from 29.6 GiB to 28.7 GiB on disk (~3% reduction).

• —Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
• —Config: compressed-tensors, num_bits=8, type=int, symmetric=true
• —Weights: INT8, symmetric, per-channel (static)
• —Activations: INT8, symmetric, per-token (dynamic)
• —Quantized: the MLA gated attention (q_proj, kv_a_proj_with_mqa, kv_b_proj, o_proj, and the sigmoid attention gate self_attn.gate_proj), the layer-0 dense MLP, and the two shared experts.
• —Kept in BF16: the 64 routed experts per layer, the MoE router (mlp.gate), lm_head, embed_tokens, and the layer norms.

The size reduction is small by design, and that is the main thing to understand about this checkpoint. The routed experts account for 26.8 GiB of the 28.7 GiB total, and they stay in BF16 because vLLM's CPU path has no INT8 MoE backend. What INT8 buys here is faster attention and dense-MLP matmuls, not a smaller memory footprint. The router is skipped for the usual reason: it is tiny and mis-routing costs far more accuracy than 8-bit weights do.

Two naming details drive the ignore list. The router module is mlp.gate (an nn.Linear), not mlp.gate_proj — that name belongs to the SwiGLU expert and dense FFN weights — so the regex anchors on mlp\.gate$. And the routed-expert pattern uses a trailing dot, mlp\.experts\., so that shared_experts does not match and stays quantizable.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model_id = "amd/Instella-MoE-16B-A3B-SFT"
output_dir = "./Instella-MoE-16B-A3B-SFT-w8a8-llmcompressor"

# Step 1: Load the BF16 model and tokenizer. Instella ships custom modeling code
# built on DeepSeek-V3, so trust_remote_code is required.
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Define the W8A8 recipe.
#   - mlp.gate is the DeepseekV3TopkRouter (64 experts, top-6): tiny and
#     extremely routing-sensitive, so skip it. Note the anchor: mlp.gate$ does
#     not match the expert/dense mlp.gate_proj weights, which stay quantizable.
#   - mlp.experts. (trailing dot) skips the routed expert FFNs while leaving
#     mlp.shared_experts quantizable. The routed experts are left in BF16
#     because vLLM CPU has no W8A8 int8 MoE backend.
recipe = QuantizationModifier(
    scheme="W8A8",
    targets=["Linear"],
    ignore=[
        "lm_head",
        r"re:.*lm_head",
        r"re:.*mlp\.gate$",
        r"re:.*mlp\.experts\.",
    ],
)

# Step 3: One-shot quantize and save in compressed-tensors format.
# W8A8 here is data-free (RTN), so no calibration dataset is needed.
oneshot(
    model=model,
    recipe=recipe,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Instella-MoE-16B-A3B-SFT-w8a8-llmcompressor",
    dtype="bfloat16",
    trust_remote_code=True,
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.13.0
zentorch==2.13.0
vllm==0.28.0
llmcompressor==0.13.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Evaluation Command

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Instella-MoE-16B-A3B-SFT-w8a8-llmcompressor,dtype=bfloat16 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

• —Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.13.0 / PyTorch v2.13.0. It may not load correctly on other versions.
• —CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
• —Routed Experts Unquantized: vLLM CPU has no INT8 MoE backend, so the routed experts remain in BF16. Memory footprint is therefore close to the BF16 model; the benefit is compute, not capacity.
• —Custom Modeling Code: The checkpoint carries Instella's own modeling files and requires trust_remote_code=True to load.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.