zeallat/Huihui-Qwen3.8-27B-abliterated-FP8-dynamic

Huihui-Qwen3.8-27B-abliterated-FP8-dynamic

FP8 (W8A8, dynamic per-token activations / per-channel weights) quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, produced with llmcompressor 0.13.0, with the MTP (multi-token prediction) head preserved so vLLM speculative decoding (--speculative-config '{"method":"mtp","num_speculative_tokens":1}') works.

What is quantized

Only the 256 Linear modules of the language model's full-attention and MLP paths are FP8. Left in BF16: the vision tower (model.visual.*), the GatedDeltaNet linear-attention layers (*.linear_attn.*), lm_head, embeddings, and the whole mtp.* head.

Recipe (data-free, no calibration set)

from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "huihui-ai/Huihui-Qwen3.8-27B-abliterated"
model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="bfloat16")
processor = AutoProcessor.from_pretrained(MODEL_ID)
recipe = QuantizationModifier(
    targets="Linear", scheme="FP8_DYNAMIC",
    ignore=["re:.*lm_head", "re:.*embed_tokens$", "re:.*visual.*", "re:.*model.visual.*",
            "re:.*linear_attn.*", "re:^mtp.*", "re:.*mtp.*"],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained("out-fp8"); processor.save_pretrained("out-fp8")

MTP graft

Qwen3_5ForConditionalGeneration does not instantiate the mtp.* submodule, so save_pretrained silently drops the 15 MTP tensors. After quantization they were copied verbatim (BF16) out of the source shards into a dedicated shard, model.safetensors.index.json was regenerated to cover them, and re:.*mtp.* was added to quantization_config.ignore. A copied-in shard without an index entry is ignored at load time — the drafter then runs on random weights and acceptance drops to ~0%.

Verification

vLLM 0.28.0, single RTX PRO 6000 Blackwell 96GB:

vllm serve <model> --max-model-len 32768 --max-num-seqs 256 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

• —Resolved architecture: Qwen3_5MTP, Detected MTP model. Sharing target model embedding weights with the draft model.
• —Greedy 300-token generation: spec_decode_num_accepted_tokens_total 141 / num_draft_tokens_total 181 = 77.9% acceptance, confirming the grafted MTP weights are real.

Note: on a 96GB card the default max_num_seqs=1024 exceeds the available Mamba cache blocks and boot fails; pass --max-num-seqs 256 (or lower) for the hybrid GatedDeltaNet layers.