Modelpublic

Avesed/Qwen3.6-27B-INT8-W8A8

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

5likes1.4kdownloads

Model Card

Qwen3.6-27B-INT8-W8A8

INT8 (W8A8) quantization of [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) — a dense hybrid model (GatedDeltaNet linear-attention + periodic full-attention) with a vision tower and an MTP head.

Quantization

—Symmetric INT8 weights (per-channel, MSE observer) + INT8 dynamic per-token activations (llm-compressor). int8 is robust, so no AWQ/SmoothQuant smoothing is needed.
—Quantized: all MLP, the full-attention self_attn, and the GatedDeltaNet out_proj.
—Kept bf16 (quality-sensitive): GatedDeltaNet in_proj_a/in_proj_b, lm_head, embeddings, vision tower, MTP head.
—Format: compressed-tensors (int-quantized). Full recipe: `recipe.yaml`.

INT8 W8A8 is near-lossless; for the smallest footprint use the INT4-W4A16 variant (GSM8K 96.8% / MMLU-Pro 82.4%).

Usage (vLLM)

bash

vllm serve Avesed/Qwen3.6-27B-INT8-W8A8 \
  --tensor-parallel-size 2 --trust-remote-code --reasoning-parser qwen3

Served via vLLM's INT8 W8A8 CUTLASS path (works on Ampere sm80 / sm86).

Quantized with vllm-ampere-optimized/quantize.