Modelpublic

casperhansen/Qwen3.6-35B-A3B-INT4-RTN

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes41downloads

Model Card

Qwen3.6-35B-A3B-INT4-RTN

INT4 (weight-only) quantization of `Qwen/Qwen3.6-35B-A3B`, produced with `llm-compressor` using RTN (round-to-nearest, data-free) quantization.

The quantization scheme is a faithful reproduction of the native INT4 scheme used by [Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) / [Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking): INT4 weight-only, group size 32, symmetric, applied only to the routed MoE experts and stored in the compressed-tensors pack-quantized format.

Note: Moonshot produced Kimi's weights with Quantization-Aware Training (QAT). This checkpoint reproduces the same scheme and on-disk format via post-training RTN — not QAT — so it is directly loadable by the same inference engines, but is not QAT-trained.

Quantization details

Property	Value
Method	RTN (round-to-nearest), data-free (`llmcompressor.model_free_ptq`)
Precision	W4A16 (4-bit integer weights, 16-bit activations)
Strategy	group, group_size = 32
Symmetric	true
Format	`pack-quantized` (compressed-tensors)
Quantized modules	routed MoE experts only (`.mlp.experts..{gate,up,down}_proj`), incl. the MTP module
Left in BF16	attention (`self_attn` + `linear_attn`/Gated DeltaNet), shared expert, router gates, vision tower, `lm_head`, embeddings, norms

This mirrors Kimi's config: num_bits=4, type=int, strategy=group, group_size=32, symmetric=true, format=pack-quantized, quantizing the MoE components while keeping attention and the shared expert at full precision.

Result: ~72 GB (BF16) → ~23 GB on disk.

Usage (vLLM)

bash

vllm serve casperhansen/Qwen3.6-35B-A3B-INT4-RTN \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --reasoning-parser qwen3
# add --language-model-only for text-only serving

python

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="casperhansen/Qwen3.6-35B-A3B-INT4-RTN",
    messages=[{"role": "user", "content": "Give me a fun fact about the ocean."}],
    max_tokens=256, temperature=0.7, top_p=0.8,
)
print(resp.choices[0].message.content)

Verified to load and generate on a single NVIDIA H200 with vLLM (INT4 WNA16 MoE Marlin path).

Reproduction

python

from compressed_tensors.config import CompressionFormat
from compressed_tensors.quantization import (
    QuantizationArgs, QuantizationScheme, QuantizationStrategy, QuantizationType,
)
from llmcompressor import model_free_ptq

scheme = QuantizationScheme(
    targets=[r"re:.*mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"],
    weights=QuantizationArgs(
        num_bits=4, type=QuantizationType.INT, strategy=QuantizationStrategy.GROUP,
        group_size=32, symmetric=True, observer="minmax", dynamic=False,
    ),
    format=CompressionFormat.pack_quantized.value,
)

model_free_ptq(
    model_stub="Qwen/Qwen3.6-35B-A3B",
    save_directory="Qwen3.6-35B-A3B-INT4-RTN",
    scheme=scheme,
    max_workers=8,
    device="cuda:0",
)

License

Apache-2.0, inherited from the base model `Qwen/Qwen3.6-35B-A3B`.