twolven/Qwen3.8-27B-abliterated-AWQ-MTP

Qwen3.8-27B-abliterated-AWQ-MTP

W4A16 AWQ quantization of the abliterated (refusal-removed) Qwen3.8-27B, with the MTP speculative-decoding module preserved.

As of 2026-08-15 this appears to be the only AWQ / W4A16 build of an abliterated Qwen3.8-27B. Other abliterated derivatives are GGUF, MLX, NVFP4, MXFP4, BF16 or INT8 — none of which give a fast 4-bit weight-only path under vLLM on Ampere. It is also one of only two abliterated builds of any format that preserve the MTP module.

To be clear about what this is not: AWQ builds of the base (non-abliterated) Qwen3.8-27B do exist and several predate this one. The gap filled here is the abliterated + W4A16 + MTP combination, not AWQ of this model in general.

• —Base: JonathanColetti/Qwen3.8-27B-Uncensored (bf16 abliteration, the only abliterated 3.8 that kept the MTP module)
• —Upstream: Qwen/Qwen3.8-27B (Apache-2.0)
• —Method: llm-compressor 0.13.0, AWQ + W4A16_ASYM, group size 128
• —Calibration: 128 samples × 1024 tokens from HuggingFaceH4/ultrachat_200k

Architecture notes

qwen3_5 is a hybrid architecture and a VLM:

Left in bf16 (not quantized): the whole vision tower (model.visual.*), lm_head, mtp.*, and in_proj_a / in_proj_b — the 48-wide per-head scalars driving the delta rule, where 4-bit is destructive and saves nothing. conv1d is not an nn.Linear so AWQ never touches it.

v_proj → o_proj smoothing is deliberately skipped: GQA means v_proj emits 1024 while o_proj takes 6144 (q_proj is doubled to 12288 by attn_output_gate), so the shapes do not line up for a channel-wise scale.

Measured on 2× RTX 3090 (TP=2, vLLM)

All figures are single-stream (c=1), 600 output tokens, thinking off, measured over an HTTP round-trip with the same harness — so they are directly comparable to each other.

Note on absolute numbers. Throughput on these builds is workload-dependent by roughly 30%: MTP acceptance is far higher on predictable text than on varied prose. Measured 2026-08-21 on the sibling HauhauCS AWQ build under a pinned protocol — same box, same settings, same day — an expository prompt gives 95.9 t/s where creative prose gives 74.3 t/s. The figures in this section were all taken in one session with one harness, so they are internally comparable; but the prompt was not recorded, so do not quote the absolute values against numbers measured elsewhere. The relative comparisons (W4A16 vs FP8, MTP on vs off) are unaffected.

What MTP is actually worth here

Measured on this model, same harness, only --speculative-config changed:

That third row is the failure in trap 3 below, and it is worth internalising: a broken MTP is slower than no MTP at all, because you pay the full drafting cost (~98 t/s of drafted tokens) and accept none of it. It presents as a mysterious throughput regression, not as a loading error. Disabling MTP entirely also buys you a larger KV cache (677,958 tokens, 2.59× at 262,144) if context depth matters more to you than latency.

Verified after quantization: 250,060-token needle retrieval; tool calling (correct JSON arguments); the abliteration survived; and vision still works — the model correctly described shapes and colours and read embedded text from a test image. The vision tower is left in bf16, so image quality should be unchanged from the source.

Not measured: no perplexity or benchmark comparison against the bf16 source was run. W4 does cost some accuracy relative to bf16/FP8; this card does not quantify it. Behaviour was verified, quality regression was not.

Recommended sampling settings — read this before you file a bug

Do not use greedy decoding (`temperature: 0`) with this model. It will emit the same sentence over and over until it hits your token cap and never produce a stop token. This is the classic Qwen3 + quantization degenerate-repetition mode, not a defect in the weights, and it is fully reproducible: greedy is deterministic, so the same prompt loops the same way every time.

generation_config.json in this repo now ships a repetition_penalty of 1.05, which is enough to prevent it. If your stack ignores generation_config.json, set it yourself:

Measured on this build, greedy worst case, 6000-token cap:

The penalty is not free — it costs about 6% throughput. Interleaved A/B/A/B on the same box, thinking disabled, 600-token generations, c=1: 68.4 / 67.1 t/s median with the penalty off versus 64.3 / 63.6 t/s at 1.05. That is the price of not looping; we think it is worth paying by default, and you can lower it to 1.02 if you would rather have the throughput.

It does not otherwise cost you anything measurable. Checked at 1.05 against penalty-off on the tasks a repetition penalty is most likely to damage — all identical:

Do not go much above 1.10 — that is where repeated code syntax and identifiers start being penalized.

Two traps if you try to fix this yourself

• —`presence_penalty` will not work as a server-side default in vLLM. vLLM only carries a fixed whitelist out of generation config into its default sampling params — repetition_penalty, temperature, top_k, top_p, min_p, max_new_tokens (ModelConfig.get_diff_sampling_param). A presence_penalty entry is silently ignored.
• —`presence_penalty` and `frequency_penalty` cannot be defaulted at all over the OpenAI API, because the OpenAI schema defaults them to 0.0 — clients always send them explicitly, so your default never applies. repetition_penalty defaults to None on the request and is filled from the server's defaults, which is why it is the only lever that survives a client setting its own temperature.

Why your client probably will not save you

Coding agents commonly pin temperature: 0 for determinism and send no penalty at all. Two checked as of 2026-08-21: one sends no temperature when thinking is enabled but hardcodes temperature: 0 on its sub-agent and skill paths; the other sends no temperature and no penalty of any kind, ever. In both cases every request rides entirely on the server defaults.

Optional: chat_template_medium.jinja

This model's stock chat template defaults reasoning_effort to xhigh, which spends the whole budget thinking and returns an empty answer on a large fraction of requests. The default chat_template.jinja in this repo is unmodified upstream — we did not silently change the behaviour of a redistributed artifact.

For convenience an opt-in copy is included as `chat_template_medium.jinja`, byte-identical except for one line, which defaults reasoning_effort to medium instead:

-{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
+{%- set resolved_reasoning_effort = reasoning_effort|default('medium') %}

Use it with --chat-template chat_template_medium.jinja, or keep the stock template and pin the value per-request with --default-chat-template-kwargs as shown below. Both work; the flag merges per-key, so clients sending their own chat_template_kwargs stay covered.

Running it

Needs a vLLM recent enough to register Qwen3_5ForConditionalGeneration and the Qwen3_5MTP proposer. This was validated on vllm/vllm-openai:nightly (v0.20.2rc1.dev129, compressed-tensors 0.15.0.1 in-image — it reads the 0.18.0 config fine).

Minimal

vllm serve twolven/Qwen3.8-27B-abliterated-AWQ-MTP \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The full production config (2× RTX 3090, single user)

This is what produced the numbers in the table above.

vllm serve /path/to/Qwen3.8-27B-abliterated-AWQ-MTP \
  --served-model-name qwen3.8-27b-abliterated \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 4096 \
  --kv-cache-dtype fp8 \
  --performance-mode interactivity \
  --mm-encoder-tp-mode data \
  --disable-custom-all-reduce \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "medium"}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode": "PIECEWISE"}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

With NCCL_P2P_DISABLE=1 in the environment.

Why these particular flags — each one is load-bearing on this hardware:

⚠️ Pin reasoning_effort — the default will truncate your answers

The chat template accepts reasoning_effort of xhigh / medium / low, and defaults to `xhigh`:

{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}

xhigh is pathological for general serving. Measured over 4 hard reasoning prompts with a 16,384-token budget:

At xhigh, half the requests spent the entire budget thinking and returned an empty answer (finish_reason: length). Note low is counterintuitively worse than medium — it thinks ~58% more — so medium is the sweet spot, not merely a midpoint.

This is not a repetition loop and not a quantization artifact. A truncated xhigh trace analysed at 675/676 unique sentences, with only 1.7% of positions inside any repeated 8-gram and 0% repetition in the final 400 words — the model is generating novel reasoning right to the cutoff. Re-running an identical truncated prompt with a 40,000-token budget terminated cleanly (stop, 11,756 thinking + 2,913 answer). The failure mode is variance: xhigh's thinking length straddles typical budgets, so the same prompt sometimes finishes at 14k and sometimes blows past 16k and returns nothing.

Pin it server-side:

--default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "medium"}'

Clients can still opt up per request with chat_template_kwargs: {"reasoning_effort": "xhigh"} — just give them a budget of 40k+ if they do. Thinking can be turned off entirely with {"enable_thinking": false}.

Tool calling caveat

If you enable --tool-call-parser qwen3_coder, vLLM's parser has a streaming bug where the <tool_call> tag is emitted as prose and subsequent SSE chunks go silent. We run a small patch that buffers <tool_call> tokens until <function= appears within 64 characters. Patch and details: club-3090 → models/qwen3.6-27b/vllm/patches/local/. Non-streaming tool calls are unaffected.

Sanity checks after it boots

# should report ~560,900 tokens and 2.14x at 262,144
grep -E "GPU KV cache size|Maximum concurrency" <container logs>

# MTP must actually load — if you see "not found in params_dict" the drafter is
# running on random weights and acceptance will be 0% (see trap 3 below)
grep "Detected MTP model" <container logs>

# after some traffic, acceptance length should be ~2.9 of 3
grep "SpecDecoding metrics" <container logs>

The compose files, the quantization script, and the deployment notes this model came out of live in [club-3090](https://github.com/noonghunna/club-3090).

If you reproduce this — four things that will bite you

1. 1.AWQ mappings must be generated per layer. match_modules_set() accumulates matches until every pattern fires, so a generic re:.*input_layernorm$ → q/k/v mapping piles up four smooth layers across three consecutive linear_attention blocks (which have no q_proj) and aborts with "AWQ needs to match a single smoothlayer". Emit one mapping per layer index — 192 over 64 layers.
2. 2.transformers erases `Qwen3_5GatedDeltaNet.forward`'s signature. Its force_accelerate_hooks decorator is def wrapped(self, *args, **kwargs) with no functools.wraps, so inspect.signature() reports (*args, **kwargs); AWQ's arg cache then stores {'args': …} and replay dies with "missing 1 required positional argument: 'hidden_states'". Recover the inner function from the wrapper closure and re-attach __signature__.
3. 3.MTP needs both an index entry and an ignore rule. save_pretrained writes a single model.safetensors with no index, so a copied-in MTP file is never opened, the drafter loads random weights, and you get 0% draft acceptance — which presents as a speed bug (~33 t/s) rather than a load bug. Write a model.safetensors.index.json covering both files, and add re:.*mtp.* to quantization_config.ignore or vLLM builds MTP as quantized and demands weight_packed. After changing that, clear the torch.compile cache or a stale AOT graph throws KeyError: 'weight_packed'.
4. 4.AWQ will OOM 24 GB at 256×2048 calibration. Use offload_device="cpu" on AWQModifier and drop to 128×1024.

Author

Quantized by Todd Wolven - Lead AI Software Developer and open-source GenAI engineer.

Other projects and writeups | GitHub | Hugging Face

License & attribution

Apache-2.0, inherited from Qwen/Qwen3.8-27B. The abliteration is JonathanColetti's; this repo contributes only the quantization.

This model is uncensored. Refusal behaviour has been removed by the upstream abliteration and that property survives quantization. You are responsible for how you use it.