tirex2001/Qwen3.6-40B-Deckard-AWQ-gemm-MTP

Qwen3.6-40B-Deckard — AWQ (GEMM) 4-bit for vLLM, with a working MTP head

### ⚡ Built for 2× RTX 2080 Ti (Turing / SM75) + NVLink A 40B reasoning model at ~42–44 tok/s with working MTP speculative decoding on a pair of 2019 gaming cards — faster than the same model's MTP GGUF in llama.cpp (~35 tok/s) on the same hardware. If you have old Turing cards gathering dust, this is for you.

The first vLLM-servable AWQ quant of DavidAU's Qwen3.6-40B Opus-Deckard that includes a working Multi-Token Prediction (MTP) speculative-decoding head. Complements PiehSoft's GGUF release (llama.cpp-only) by bringing MTP to the vLLM / AWQ path.

Reference hardware (what all numbers below were measured on)

Stock 11 GB 2080 Tis will not fit this model in TP2 (~17.6 GiB weights per GPU). 2× 22 GB modded cards, or any pair with ≥20 GB per card and SM ≥ 7.5, is the realistic floor.

What this is

• —Base: DavidAU/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking — a 27B→40B width-preserving layer expansion (hidden 5120, 96 decoder blocks: 72 GatedDeltaNet linear-attention + 24 full-attention, interval 4). Reasoning ("Thinking") model; uncensored ("Heretic") finetune — use responsibly.
• —Quantization: 4-bit AWQ, `version: gemm` (AutoAWQ export layout), served with vLLM's awq_marlin kernel. Produced with Intel AutoRound 0.14.2 in RTN mode (iters=0, no gradient rounding optimization — see Honesty notes). bits=4, group_size=128, sym=False (asymmetric, runtime zero-points).
• —MTP head: transplanted from a Qwen3.6-27B donor (QuantTrio/Qwen3.6-27B-AWQ-6Bit), kept in BF16 (15 tensors, 0.85 GB, model-mtphead.safetensors). The 40B expansion preserved the 27B's width — hidden 5120, intermediate 17408, 24 attention heads / 4 KV heads, headdim 256, vocab 248320 are **identical** — so the donor head drops in unchanged. Same transplant idea as PiehSoft's `injectmtp_40b.py`, done in the safetensors/HF path. The head was not fine-tuned on the 40B.

Precision layout (what is 4-bit vs what stays BF16)

GDN in-projections and attention q/k/v are deliberately kept in BF16: quality of the hybrid linear-attention state and draft acceptance of the MTP head both degrade when they are quantized. (Quantizing GDN in_proj also turns out to save disk only — vLLM's GatedDeltaNet kernel dequantizes them back to fp16 in VRAM, so there is no context-length benefit. Measured, not guessed.)

Measured performance (single stream, greedy unless noted)

• —Methodology: 520-token generations with ignore_eos, decode rate computed as a delta between a long and a short run (excludes prompt/TTFT). TTFT on short prompts ≈ 0.4 s.
• —MTP draft acceptance (transplanted, untuned head): ≈78% average on greedy fresh-context; per-position ≈ 0.88 / 0.60 / 0.42 at depth 3; mean acceptance length ≈ 2.7–2.9 tokens per step.
• —Comparison point: the same model's MTP GGUF in llama.cpp (NCCL build, -sm tensor, NVLink) does ~35 tok/s on the same cards. This vLLM path is ~20% faster single-stream, and unlike llama.cpp it batches concurrent requests.
• —Why contexts differ: weights take ~17.6 GiB/GPU, leaving ~1.9 GiB for KV at gpu_memory_utilization 0.94 → ~27k max without MTP; the safe-sync MTP path reserves ~1 GiB more → ~12k. KV-cache fp8 does not engage on SM75; vLLM's --kv-offloading-size is a prefix-cache offload and does not extend a single request's context.

⚠️ Two SM75 pitfalls (read before serving)

1. 1.Never set `VLLM_ALLOW_MAMBA_SPEC_FULL_CUDAGRAPH=1` with MTP on this model. Full-CUDA-graphing the GatedDeltaNet (linear-attention) state update during speculation replays stale recurrent state and silently corrupts generation — repetition loops, hallucinated prompts, empty outputs — while short trivial prompts can still look fine, which makes it easy to miss. Keep the default 0, and use VLLM_SM75_SPEC_SYNC_MODE=safe (not nosync, which corrupts the same way). Symptom → cause was verified by A/B on identical prompts.
2. 2.`--disable-custom-all-reduce` is required on this 2× 2080 Ti setup: the custom all-reduce kernel fails with CUDA graphs (custom_all_reduce.cuh: invalid argument). NCCL over NVLink takes over the TP all-reduce; cost is ~2 tok/s.

Serving — recommended (MTP, ~42–44 tok/s, 12k ctx)

VLLM_SM75_SPEC_SYNC_MODE=safe VLLM_ALLOW_MAMBA_SPEC_FULL_CUDAGRAPH=0 \
python -m vllm.entrypoints.openai.api_server \
  --model <this-repo> --served-model-name qwen40b-deckard \
  --dtype half --tensor-parallel-size 2 --disable-custom-all-reduce \
  --quantization awq_marlin --max-model-len 12288 \
  --gpu-memory-utilization 0.94 --max-num-seqs 1 \
  --language-model-only --skip-mm-profiling \
  --mamba-cache-mode align --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[4],"max_cudagraph_capture_size":4}'

Notes:

• —CUDA-graph capture size must be a multiple of `num_speculative_tokens`+1 (here 4), or the engine aborts with "No valid cudagraph sizes".
• —Plain mode (24k ctx, ~24 tok/s): drop --speculative-config, use capture size 1 and --max-model-len 24576.
• —Long context (131k, ~3 tok/s): add --cpu-offload-gb 8 --max-model-len 131072.
• —This is a vision-language checkpoint architecture served text-only (--language-model-only); no vision weights are shipped.
• —The base model is a thinking model: responses arrive as reasoning_content + content (hence --reasoning-parser qwen3). Give it generous max_tokens.

Reproducing the quant (outline)

1. 1.Quantize the BF16 base with AutoRound (bits=4, group_size=128, sym=False, iters=0, format="auto_awq"), keeping the BF16 list above via layer_config={name:{"bits":16}}. Two patches are needed for this VL-architecture checkpoint: force the LLM (not MLLM) calibration path, and fix the checkpoint-name reverse mapping so tensors export as model.language_model.* (matching how vLLM loads the composite arch).
2. 2.Transplant the MTP head: copy the 15 mtp.* tensors from the 27B donor into a separate safetensors shard and register them in model.safetensors.index.json. No renaming needed — mtp.* lives top-level.
3. 3.Config: base model's config.json + quantization_config {quant_method: awq, bits: 4, group_size: 128, version: gemm, zero_point: true, modules_to_not_convert: [...]} (see this repo's config for the exact list). mtp_num_hidden_layers: 1 is already present in the base config.

Honesty notes / limitations

• —The quant is RTN (round-to-nearest, no per-block gradient optimization). It measures coherent across RU/EN chat, code, and reasoning prompts, but a calibrated AutoRound run (iters=200) would likely be marginally more accurate. May follow later.
• —The MTP head is a dimensional transplant, not a trained one — acceptance (~78% greedy) is good but below a natively trained head; creative/high-temperature text will accept less and gain less speed.
• —The base model is a community franken-expansion of Qwen3.6-27B and an uncensored finetune. Quality characteristics of the base are inherited as-is; the base model's license and usage terms apply.
• —Context ceilings above are specific to 2× 22 GB cards; more VRAM → linearly more KV/context.

Credits

• —DavidAU — the base Opus-Deckard 40B model.
• —QuantTrio — the Qwen3.6-27B AWQ whose MTP head was transplanted.
• —PiehSoft — the MTP-transplant method (inject_mtp_40b.py) and acceptance write-up for the GGUF path.
• —Intel AutoRound — the quantization toolkit.
• —weicj / vLLM-2080ti-definitive — the SM75 vLLM build that makes any of this possible on Turing.