tacodevs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-FP8

Qwen3.6-27B-Fable-Fusion-711 Uncensored-Heretic FP8 (vision-preserving)

Calibrated FP8 quantization of DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP that preserves vision-language capabilities, unlike vLLM's dynamic FP8 which destroys them on this architecture family.

TL;DR

• —~30 GB FP8 (down from ~52 GB BF16), serves on a single 48 GB+ GPU with 16k context.
• —Vision verified intact after quantization (correctly reads fine image details — eye color, clothing trim, props — on real multimodal workloads).
• —Do NOT pass `--quantization fp8` to vLLM — quantization config is baked in via compressed-tensors.
• —For structured/non-reasoning tasks, pass `chat_template_kwargs: {"enable_thinking": false}` — see Usage. Without it the model emits long tag-less "Thinking Process:" reasoning before answering.

About the source model

The source is DavidAU's multi-stage merge on the Qwen3.6-27B base (Qwen qwen3_5 architecture class: hybrid attention with Gated DeltaNet linear-attention layers + full attention every 4th layer, vision encoder included, 256k native context). Decensoring is via Heretic v1.2 with Arbitrary-Rank Ablation (ARA) — the source card measures 4/100 refusals vs 99/100 for the base model.

Note: the source repo ships a sidecar MTP (multi-token prediction) tensor file that is not part of the model index. It is not included in this quant; vLLM does not use it for standard serving.

Why dynamic FP8 destroys vision on this architecture

vLLM's runtime --quantization fp8 uses a single tensor-wide scale per Linear layer. The vision merger — the only bridge between the 1152-dim visual tower and the 5120-dim LM embedding space — has a much wider weight distribution than LM layers, so single-scale FP8 rounds its small-magnitude weights to zero. The LM then receives noise at image-token positions and silently hallucinates image descriptions from text alone. See the detailed write-up in the sibling repo tacodevs/Huihui-Qwen3.5-27B-Claude-4.6-Opus-abliterated-FP8.

What this checkpoint does differently

• —Per-channel weight scales (one scale per output channel, computed from actual weight distributions) instead of one global scale per layer.
• —Entire visual tower and merger kept in BF16 via the ignore list.
• —All Gated DeltaNet `linear_attn` modules kept in BF16 (hybrid-attention internals are not plain GEMMs and are excluded).
• —lm_head kept in BF16.
• —Dynamic per-token activation quantization at inference time.

Only the language model body (full-attention projections and MLP linears) is FP8.

Quantization details

• —Tool: llmcompressor (see recipe.yaml in this repo)
• —Scheme: FP8_DYNAMIC (per-channel weight scales, dynamic per-token activation scales)
• —Targets: all Linear layers
• —Excluded modules (207 total): re:.*visual.* (visual tower + merger), all linear_attn modules and norms, lm_head
• —Original size: ~52 GB BF16 → FP8 size: ~30 GB

Measured performance (RTX PRO 6000 Blackwell 96 GB, vLLM 0.26)

• —Decode: ~49 tok/s single-stream warm (identical to the BF16-recipe sibling Qwen3.5-27B FP8 on the same GPU).
• —TTFT ~0.8–1.0 s on multimodal requests (one 900px image + ~1k text tokens).
• —Vision: correct fine-grained image reading across all test runs (hair/eye color, clothing details, held objects, background).

Usage with vLLM

python -m vllm.entrypoints.openai.api_server \
    --model tacodevs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-FP8 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code

IMPORTANT: Do NOT pass --quantization fp8. The model already has its quantization config baked in via compressed-tensors; vLLM detects and uses the proper FP8 path automatically. Passing --quantization fp8 would re-quantize the already-FP8 weights and break everything.

Controlling thinking mode

The chat template supports Qwen's enable_thinking switch. By default the model produces extended reasoning without <think> tags (plain "Thinking Process:" markdown), which is easy to overrun token budgets with and hard to strip in streaming pipelines. For structured-output or latency-sensitive tasks, disable it per request:

{
  "model": "tacodevs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-FP8",
  "chat_template_kwargs": {"enable_thinking": false},
  "messages": [...]
}

In our structured image-prompt workload this cut output from a truncated 500+ tokens to a complete 130–210 tokens and total latency from ~11 s to ~4.7 s, with format-perfect results.

Credits

• —Source merge: DavidAU
• —Decensoring: Heretic v1.2 (Arbitrary-Rank Ablation)
• —Base model: Qwen/Qwen3.6-27B
• —Quantization: tacodevs