Modelpublic

Ar4ikov/Qwen3.8-27B-AWQ-W4A16-ASYM-HyperQwen-fast

sourceHugging Faceapache-2.0updated 3d agoView on Hugging Face

0likes74downloads

Model Card

Qwen3.8-27B-AWQ-W4A16-ASYM-HyperQwen-fast

Qwen3.8-27B-AWQ-W4A16-ASYM prepared for HyperQwen — Qwen3.8-27B served fast on one 24 GB card — the way the TurboQwen image expects it. The body is untouched: int4 asymmetric AWQ, group 128, zero points, quantized with llm-compressor from bf16 weights; the vision tower, the SSM gate projections and the MTP head's norms stay bf16. What changed is what HyperQwen's prepare/ pipeline changes, so that a 24 GB card has room for a KV cache and speculative decoding has something small to score:

tensor	in the source export	here
`lm_head`	bf16, 2.5 GB	int4 GPTQ (group 128, symmetric), calibrated on 400k of the model's own final hidden states (`drafter/gptq_lm_head.py`; RTN int4 KL 0.0070 -> GPTQ 0.0024 against the bf16 head)
`embed_tokens`	bf16, 2.5 GB	int8 (group 128, symmetric), round-trip error 0.65%
MTP module (`mtp.*`, 8 linears incl. `mtp.fc`)	bf16, 850 MB	int8 (group 128, symmetric)
`mtp.draft_lm_head` + `mtp_draft_vocab_ids.pt`	—	40,960-row draft head (int4, sliced from the int4 `lm_head`), HyperQwen's shipped id list
body (64 layers), vision tower	int4 asym g128 / bf16	unchanged

Qwen3.8-27B-AWQ-W4A16-ASYM-HyperQwen keeps lm_head in int8 instead: +0.6% perplexity less, a few tok/s less.

Serving

The container does everything (download, verify, serve), with the vision tower on:

bash

git clone https://github.com/Ar4ikov/TurboQwen && cd TurboQwen
cp .env.example .env            # CHECKPOINT=base   and MODEL=/app/models/Qwen3.8-27B-AWQ-W4A16-ASYM-HyperQwen-fast
docker compose --profile single up -d

Bare metal, on the fork branch this was measured with (Ar4ikov/HyperQwen@awq-asym, vLLM 0.29.0 + the series + marlin-int8-asym-zp):

bash

hf download Ar4ikov/Qwen3.8-27B-AWQ-W4A16-ASYM-HyperQwen-fast --local-dir models/Qwen3.8-27B-AWQ-W4A16-ASYM-HyperQwen-fast
VISION=1 MODEL=$PWD/models/Qwen3.8-27B-AWQ-W4A16-ASYM-HyperQwen-fast SPEC=mtp CTX=fast bash single-user/start_qwen.sh

Plain vLLM 0.29 also loads it (vllm serve Ar4ikov/Qwen3.8-27B-AWQ-W4A16-ASYM-HyperQwen-fast --max-model-len 65536), without the speculative decoding, the draft head or the int8 Marlin path that need the patch series.

Measured

RTX 3090 (350 W), vLLM 0.29.0, HyperQwen bench/run_benchmarks.sh single, second run after boot, VISION=1 (the tower streamed from host RAM per image). C1 = one stream of real prompts with 1,024-token answers; tok/step = tokens accepted per forward pass.

profile	C1 T=default	C1 T=0	tok/step	C8 T=default	KV pool
`SPEC=dflash2 CTX=fast KV_MEM=4600000000 DFLASH_MAX_LEN=49152`	130.7 tok/s	144.9	3.17 / 3.44	449 tok/s	53,233
`SPEC=mtp CTX=fast` (64k)	not run separately; the int8-head sibling measures 111.4 / 116.0 and the uncensored fast variant gains +5% / +9% over its sibling

Images are described correctly in every profile (a drawn red square, blue circle and a line of text: boost/image_smoke.py). The whole table, the int8 (W4A8) profiles and the kernel measurements: github.com/Ar4ikov/TurboQwen.

Why a separate repo

HyperQwen's pipeline rewrites the checkpoint in place (int8 heads, the draft head) and its launchers, verify script and Docker entrypoint expect that layout. Doing it once and publishing the result turns a ~10-minute CPU step per machine into a download, and keeps the original export as it is for transformers and plain vLLM users.

Quantization recipe of the body, calibration set and the AWQ mappings for the hybrid Gated-DeltaNet / attention layers: see the source export's card and its recipe.yaml (kept here). Root model: Qwen/Qwen3.8-27B, Apache-2.0.