Modelpublic

leminkozey/Qwen3.8-27B-Uncensored-W4A16-AutoRound

sourceHugging Faceupdated 9d agoView on Hugging Face

6likes3.5kdownloads

Model Card

Qwen3.8-27B-Uncensored-W4A16-AutoRound

An AutoRound quantization of JonathanColetti/Qwen3.8-27B-Uncensored, built to run on a single 24 GB GPU with HyperQwen.

I followed syv-ai's quantization recipe using the uncensored checkpoint as the base. Credit for the recipe, vLLM patches, and serving setup goes to the HyperQwen team.

—Body: W4A16 AutoRound, group size 128.
—LM head and embeddings: INT8 using HyperQwen's quant_lm_head.py and quant_embed.py.
—Vision, MTP, and `in_proj_a` / `in_proj_b`: kept in floating point during AutoRound quantization.

Tested on a single RTX 3090 with HyperQwen's DFlash2 setup: around 98–110 tokens/s after warmup, with a configured context limit of 245,760 tokens. Retrieval was tested at 45k context, with coherent German and English output. These are results from my setup, not a full benchmark.

I tested the DFlash2 path. Another user reported working MTP in the original issue.

For installation, runtime patches, and serving instructions, see [syv-ai/HyperQwen](https://github.com/syv-ai/HyperQwen).