GotoAI-Inc/Qwen3.6-27B-W4A16

Qwen3.6-27B-W4A16

Int4 weight-only quantization of Qwen/Qwen3.6-27B, in compressed-tensors format for vLLM. 19.42 GB, down from 55.56 GB — it fits a 24 GB card, with the KV budget discussed under Usage.

Unofficial and unaffiliated with Alibaba/Qwen. All model capabilities, evaluations and limitations belong to the original model card — see the base model for those.

What was changed

Weights were quantized from bfloat16 to int4, group size 128, symmetric, weight-only (activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was used and the model was never loaded — the quantizer operates directly on the safetensors. Architecture, tokenizer, chat template and processor configs are the vendor's, unmodified.

496 Linear modules were converted, covering 64.7% of the checkpoint's bytes:

Four things are deliberately left at 16-bit:

• —*`model.visual.** — vLLM builds multimodal towers with quant_config=None`, so a checkpoint carrying quantized vision weights cannot be loaded.
• —*`mtp.** — the built-in multi-token-prediction speculator head (mtpnumhidden_layers: 1`), loaded through vLLM's speculative-decoding path rather than the main stack.
• —`linear_attn.conv1d` — 3-D causal-convolution kernels in the gated-DeltaNet blocks, shape (10240, 1, 4). Not Linear layers, and quantizers reject them outright.
• —`lm_head` + `embed_tokens` — precision-sensitive, and lm_head is untied here.

The gated-DeltaNet projections (in_proj_qkv, in_proj_a, in_proj_b, in_proj_z, out_proj) are quantized; only the convolution kernels beside them are excluded, along with the 1-D A_log and dt_bias state-space parameters, which any quantizer skips automatically.

Usage

Runs on released vLLM. Qwen3.6 reuses the Qwen3.5 architecture (Qwen3_5ForConditionalGeneration, model_type: qwen3_5), which has been supported since 0.25.1 — no nightly build is required:

vllm serve GotoAI-Inc/Qwen3.6-27B-W4A16 \
  --max-model-len 65536 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Do not pass --quantization; compressed-tensors is detected from config.json. The int4 W4A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.

• —`--tool-call-parser qwen3_coder` is what the base model card specifies. In current vLLM qwen3_xml is an alias for the same parser class, so either name works; without one, the <tool_call><function=…><parameter=…> XML the chat template asks for is returned as plain text.
• —`--reasoning-parser qwen3` splits <think>…</think> into reasoning_content.
• —`--language-model-only` skips the vision tower and its multimodal profiling, freeing ~0.92 GB of weights plus the profiling headroom, at the cost of image and video input.
• —MTP speculative decoding uses the head already in this checkpoint — no draft model to download: --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'. The base card writes "method": "qwen3_next_mtp"; current vLLM deprecates the per-family names and normalizes them to mtp, resolving qwen3_5 to Qwen3_5MTP from the checkpoint's own config. vLLM aligns the draft's quantization with the target's, and the re:.*mtp.* entry in this checkpoint's ignore list keeps those tensors bf16. Not smoke-tested here.

Fitting the card

Only 16 of the 64 layers use full attention (every 4th; the other 48 are gated DeltaNet with constant-size recurrent state). With 4 KV heads at head_dim 256, the KV cache costs ~64 KB/token — about 2 GB at 32k and 4 GB at 64k. Against 19.42 GB of weights that is comfortable on 32 GB and above; on a 24 GB card, budget for roughly 32k of context, or add --language-model-only for more. This is arithmetic from config.json, not a measured deployment.

Controlling thinking

Qwen3.6 thinks by default and does not support the /think and /nothink soft switches. It also has no reasoning_effort knob — the two template variables it does accept are passed through chat_template_kwargs:

{"chat_template_kwargs": {"enable_thinking": false}}     // instruct / non-thinking mode
{"chat_template_kwargs": {"preserve_thinking": true}}    // keep earlier turns' thinking

preserve_thinking is the feature this release adds: by default only the thinking from the latest user message is retained, and turning it on keeps historical reasoning traces in context — the base model card recommends it for agentic use, where it improves decision consistency and KV-cache reuse. Set a server-wide default with --default-chat-template-kwargs '{"preserve_thinking": true}'; request-level values still win.

Sampling, per the base model card: temperature=1.0, top_p=0.95, top_k=20 for thinking mode, temperature=0.6 for precise coding, and temperature=0.7, top_p=0.80, presence_penalty=1.5 in non-thinking mode.

Context

262144 tokens natively. The base model card documents a YaRN recipe reaching 1,010,000 tokens via --hf-overrides plus VLLM_ALLOW_LONG_MAX_MODEL_LEN=1; rope_type is left at default here, and static YaRN costs quality at short contexts, so enable it only if you need it.

Reproducing this checkpoint

Built with llm-quantizer:

./llmq.py run --profile qwen3.6-27b

which is equivalent to:

# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,
# transformers==5.15.1, torch==2.13.0
from llmcompressor import model_free_ptq

model_free_ptq(
    model_stub="Qwen/Qwen3.6-27B",
    save_directory="Qwen3.6-27B-W4A16",
    scheme="W4A16",
    ignore=["re:.*visual.*", "re:.*mtp.*", "re:.*\\.conv1d$",
            "lm_head", "re:.*embed_tokens.*"],
    device="cuda:0",
)

The source ships as 15 shards of ~4 GB, and a job holds one shard at a time, so the build peaks at a few GB of VRAM — no re-sharding needed and no large GPU required.

Evaluation

No benchmarks have been run. Data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured here. Treat the published Qwen3.6 numbers as describing the bfloat16 model, not this one.

For an agentic model the informative checks are well-formed reasoning_content and clean multi-step tool calls rather than perplexity: structured emission degrades before fluency does.

License

Apache 2.0, inherited from the base model — the vendor's LICENSE is included unmodified. "Qwen" is Alibaba's mark; this repository is not endorsed by or affiliated with Alibaba.