Modelpublic

amd/Qwen3.6-35B-A3B-w8a8-llmcompressor

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

1likes850downloads

Model Card

Qwen3.6-35B-A3B-w8a8-llmcompressor-v0.12.0

Model Overview

—Model Architecture: Qwen3_5MoeForConditionalGeneration
—Input: Text
—Output: Text
—Source Model: Qwen3.6-35B-A3B
—Supported Hardware: AMD EPYC (CPU inference)
—Preferred Operating System: Linux
—Inference Engine: vLLM v0.26.0
—Quantization Framework: LLM Compressor v0.12.0
—Quantization Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
—Compatible Stack:
—ZenDNN v6.1.0
—ZenTorch v2.11.0.3
—PyTorch v2.11.0
—LLM Compressor v0.12.0
—vLLM v0.26.0

This is a quantized version of Qwen3.6-35B-A3B created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Qwen3.6-35B-A3B using LLM Compressor via the Round-to-Nearest (RTN) algorithm. This reduces the model weights from 67.0 GiB to 35.1 GiB on disk (~48% reduction).

—Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
—Config: compressed-tensors, num_bits=8, type=int, symmetric=true
—Weights: INT8, symmetric, per-channel (static)
—Activations: INT8, symmetric, per-token (dynamic)
—Kept in BF16: MoE router (mlp.gate), shared_expert_gate, linear-attention modules, the visual tower, and lm_head. The routed expert projections are quantized.

python

import torch
from transformers import AutoProcessor, AutoTokenizer, Qwen3_5MoeForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model_id = "Qwen/Qwen3.6-35B-A3B"
output_dir = "./Qwen3.6-35B-A3B-w8a8-llmcompressor-v0.12.0"

# Step 1: Load the BF16 model and tokenizer.
# Load the top-level Qwen3_5MoeForConditionalGeneration rather than AutoModelForCausalLM,
# which would demote config.json to the text-only Qwen3_5MoeTextConfig and produce a
# checkpoint vLLM rejects.
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Define the W8A8 recipe
recipe = QuantizationModifier(
    scheme="W8A8",
    targets=["Linear"],
    ignore=[
        r"re:.*lm_head",
        r"re:.*mlp.gate$",           # Qwen MoE router
        r"re:visual.*",
        r"re:model.visual.*",
        r"re:.*embed_tokens$",
        r"re:.*shared_expert_gate$",
        r"re:.*linear_attn.*",
    ],
)

# Step 3: One-shot quantize and save in compressed-tensors format
oneshot(
    model=model,
    recipe=recipe,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# oneshot does not save the processor; multimodal checkpoints need it for vLLM.
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
processor.save_pretrained(output_dir)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

python

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Qwen3.6-35B-A3B-w8a8-llmcompressor-v0.12.0",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.11.0
zentorch==2.11.0.3
vllm==0.26.0
llmcompressor==0.12.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

bash

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Benchmark	BF16 Baseline	W8A8 (this model)	Recovery
GSM8K (5-shot)	0.9629	0.9629	100%

Evaluation results will be updated after benchmarking.

Evaluation Command

bash

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Qwen3.6-35B-A3B-w8a8-llmcompressor-v0.12.0,tokenizer=Qwen/Qwen3.6-35B-A3B,dtype=bfloat16,max_model_len=4096,language_model_only=True,enable_thinking=False \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

—Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.11.0.3 / PyTorch v2.11.0. It may not load correctly on other versions.
—CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.