voska/Qwen3.8-27B-abliterated-FP8-MTP

Qwen3.8-27B-abliterated-FP8 (MTP preserved)

FP8 W8A8 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, with the multi-token-prediction (MTP) head intact so vLLM speculative decoding actually works.

Model Overview

Why this exists

Qwen3.8-27B ships an MTP head worth roughly 2.5x tokens per forward pass under vLLM speculative decoding.

Most conversions lose it silently. Qwen3_5ForConditionalGeneration does not instantiate the MTP module, so from_pretrained() discards all 15 mtp.* tensors as unexpected keys and save_pretrained() never writes them back. Anything round-tripped through transformers — including several abliterated re-uploads on the Hub — comes out with 1184 tensors instead of 1199 and no speculation available, with nothing in the logs to say so.

This build keeps them, and declares them so vLLM can load them (that declaration is a second, separate trap — see Gotchas).

Deployment

Use with vLLM

vllm serve voska/Qwen3.8-27B-abliterated-FP8-MTP \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 8 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --trust-remote-code

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

completion = client.chat.completions.create(
    model="voska/Qwen3.8-27B-abliterated-FP8-MTP",
    messages=[{"role": "user", "content": "Explain speculative decoding in two sentences."}],
    temperature=1.0, top_p=0.95, extra_body={"top_k": 20},
    max_tokens=2048,
)
msg = completion.choices[0].message
print(msg.reasoning)   # chain-of-thought, separated by --reasoning-parser
print(msg.content)     # the answer

Three flags are not optional

vLLM defaults all three off. If you are migrating from llama.cpp — which handles all three natively — you will ship regressions without noticing. Both of these bit us in production.

`--reasoning-parser qwen3` — without it the chain-of-thought lands in content with a stray closing tag: "We need to answer user: ... Final: 4.\n</think>\n\n4". Affects every reply.

`--enable-auto-tool-choice --tool-call-parser qwen3_xml` — without these, any request with tool_choice: "auto" fails with HTTP 400 ("auto tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set"). Plain chat keeps working, so health checks and manual testing sail right past it while every agentic client is hard-broken. qwen3_xml is correct because this template emits the XML-nested form, not Hermes JSON — verify with grep -c '<function=' chat_template.jinja.

`--speculative-config` — this is what activates the MTP head. Omit it and you get a perfectly good FP8 model with the head sitting unused.

For text-only serving add --limit-mm-per-prompt '{"image":0,"video":0}'.

Verify speculation is actually working

Speed alone will not tell you:

curl -s localhost:8000/metrics | grep spec_decode_num_accepted_tokens_total

If it reads 0.0 while num_draft_tokens_total climbs, the head is loaded but every draft is rejected — you are paying for speculation and getting nothing back.

Recommended sampling parameters

Inherited from the base model; these matter and the defaults are not them.

⚠️ `min_p` is silently ignored when speculation is enabled. vLLM warns at startup: "min_p and logit_bias parameters won't work with speculative decoding." Since the recommended min_p is 0.0 this costs nothing here, but do not rely on a non-zero min_p with --speculative-config set.

Reasoning effort

The chat template accepts reasoning_effort — `xhigh` (default), `medium`, `low`:

extra_body={"chat_template_kwargs": {"reasoning_effort": "low"}}

low keeps thinking brief and moves directly to a conclusion; xhigh asks the model to validate assumptions and consider alternatives. The difference is negligible on easy prompts and grows with task difficulty.

Prompt format

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

The template opens the thinking block itself, so the model emits reasoning and then a closing tag — which is why a reasoning parser is required rather than optional.

Hardware requirements

Weights are 31.2 GB. KV cache costs 64 KB/token — low for a 27B because only 16 of 64 layers keep a full KV cache. Approximate pool at --gpu-memory-utilization 0.92:

Performance

Measured on 1x RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB GDDR7, SM120), vLLM 0.26.0, speculation at k=3.

Important caveat: this card ran at a 260 W power cap, pinning the SM clock near 952 MHz against a 3090 MHz maximum (~31% of rated clock, memory near full speed). Decode is bandwidth-bound and suffers little; prefill is compute-bound and heavily throttled here. On an unconstrained card, expect prefill materially higher.

Aggregate scaling is near-linear — 67 / 117 / 234 / 427 tok/s at 1 / 2 / 4 / 8 concurrent, zero errors at every level; per-stream throughput falls only ~17% from 1 to 8 users.

Speculation

Per-position acceptance at k=3 is 70% / 45% / 28%, so the third slot still earns its keep. Code accepts more readily than prose, as expected. These were measured with light concurrent traffic present; treat them as indicative rather than laboratory-clean.

How we actually run it

Served to a small group of concurrent users behind an OpenAI-compatible gateway, one dedicated GPU, systemd-managed container:

• —`--max-num-seqs 8` matched to real user count rather than left at default
• —`--enable-chunked-prefill` is doing heavy lifting. This replaced a llama.cpp deployment where one user pasting a 130K-token context monopolised the GPU and starved everyone else's decode to single-digit tok/s. Chunked prefill with decode priority is most of why per-stream throughput barely moves as concurrency rises.
• —`--enable-prefix-caching` for large shared system prompts across agent turns
• —k=3, not higher. vLLM warns that num_speculative_tokens > 1 runs multiple forward passes over the same single MTP layer, and acceptance decays per position. Deeper does not pay here, and published reports suggest shallower is better again at very long context.
• —Bound your restarts. A crash-looping vLLM container will happily accumulate thousands of restarts overnight while looking "up" from outside.

Migration note: llama.cpp's -c is a total KV budget divided across -np slots, whereas vLLM's --max-model-len is per-request against a shared paged pool. The pool here is smaller than a naive slots×context figure but far better utilised.

Evaluation

What has been verified:

• —Refusal parity — a 10-prompt probe spanning adult fiction, profanity, contested political argument, harm reduction, security education, blunt medical advice and dark fiction returned 0/10 refusals, identical to the BF16 source. Abliteration survives quantization.
• —Throughput and speculation — tables above.
• —Tool calling — verified end-to-end, streaming and non-streaming, with correct delta reassembly through an OpenAI-compatible gateway.

What has not been measured: standard-benchmark accuracy with a recovery percentage against the BF16 base (GSM8K / MMLU-Pro / IFEval in the style of RedHatAI's quant cards). This is the honest gap in this card. What can be said from published work is that FP8 is a lateral move from 8-bit GGUF rather than a step down — KL-divergence measurements for this model family put Q8_0 at 0.00064 against BF16 with all 8-bit formats clustered at 0.0006–0.0007, while the best 4-bit sits at 0.00835, an order of magnitude worse. Those are third-party numbers, not ours.

If you run benchmarks on this, please open a discussion and they will be added with credit.

Creation

FP8 dynamic requires no calibration data — weights quantized statically per channel, activations dynamically per token at runtime:

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head", "re:.*visual.*"],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained(DST, save_compressed=True, save_original_format=False)

Roughly 20 seconds on a free GPU, then the MTP graft (below).

Package versions used:

llmcompressor        0.13.0
compressed-tensors   0.18.0
transformers         5.14.1
torch                2.13.0
vllm                 0.26.0   (serving)

Gotchas if you build your own

1. `save_pretrained` crashes on transformers 5.14.1. You get ValueError: dictionary update sequence element #0 has length 1; 2 is required, reported misleadingly as a weight-conversion/offloading failure. Upstream cause is misplaced braces: weight_map.update({k: ...} for k in ...) builds a generator of one-key dicts instead of a dict comprehension. Pass `save_original_format=False`.

*2. The MTP head needs grafting and declaring. After quantizing, copy the 15 `mtp. tensors from the BF16 source (~0.85 GB, fine to leave unquantized) — **and add their 8 Linear modules to quantization_config.ignore`**:

mtp.fc
mtp.layers.0.mlp.{down,gate,up}_proj
mtp.layers.0.self_attn.{q,k,v,o}_proj

Skip that second step and vLLM expects weight_scale tensors that do not exist. It loads without complaint and acceptance drops to exactly zero — measured here as 7,194 draft tokens generated, 0 accepted, with decode falling from ~79 to 30 tok/s. It reads as "disappointing" rather than "broken," which is what makes it dangerous.

Bias, Risks, and Limitations

This is an uncensored model. Refusal behaviour has been ablated by the upstream abliteration. It ships no safety guardrails and will engage with material its instruction-tuned parent declines. Abliteration is not surgical — it also removes safety behaviours you may actively want, and can degrade instruction-following in ways unrelated to refusals. It inherits every bias of the Qwen3.8-27B base on top of that.

Deploy it behind your own filtering if you are exposing it to anyone but yourself. You own what you deploy.

Technical limitations:

• —Vision is untested. The tower is present and left at BF16, but this build was only exercised for text. If you need multimodal, validate it yourself first.
• —Long-context behaviour was verified for KV allocation and prefill throughput, not retrieval quality at depth.
• —No standard-benchmark accuracy numbers yet (see Evaluation).
• —Benchmarks come from one power-limited card; absolute numbers will differ on yours.

License and use

Apache 2.0, inherited from Qwen/Qwen3.8-27B through huihui-ai. You are responsible for complying with applicable law in your jurisdiction.

Credits

• —Qwen — base model
• —huihui-ai — abliteration
• —llm-compressor and vLLM — quantization and serving

Citation

@misc{qwen38_27b_abliterated_fp8_mtp,
  title  = {Qwen3.8-27B-abliterated-FP8 (MTP preserved)},
  author = {voska},
  year   = {2026},
  note   = {FP8 W8A8 quantization of Huihui-Qwen3.8-27B-abliterated with the
            multi-token-prediction head preserved for vLLM speculative decoding},
  url    = {https://huggingface.co/voska/Qwen3.8-27B-abliterated-FP8-MTP}
}