vincespeed/Qwen3.8-27B-No-MTP-GGUF-Dflash2-Kquant

https://www.aihublocal.com/llm-runner-aio

Qwen3.8-27B Q3-Kmix GGUF (MTP-Free, Pure K-Quant Family)

Four pure K-quant GGUF variants of Qwen3.8-27B-UD, regenerated from official BF16 checkpoints. All variants are MTP-free, enforce a Q2K minimum floor, and contain **no importance-matrix (IQ\*) types** — every weight is a standard K-quant (`Q2K / Q3K` / `Q4K) or f32`.

📊 Variants

All variants: 851 tensors, qwen35 hybrid SSM + Attention architecture, block_count=64, nextn_predict_layers=0.

🎯 Design Goals

1. 1.Pure K-Quant — no IQ* types, no imatrix dependency: deterministic, reproducible quantization with standard llama.cpp kernels.
2. 2.Regenerated from Official BF16 Checkpoints — no cumulative requantization error; all weights quantized once, directly from BF16.
3. 3.MTP-Free — the extra MTP head block (blk.64) is stripped; block_count=64, nextn_predict_layers=0.
4. 4.Q2_K Minimum Floor — no tensor falls below Q2K (2.96 BPW); ultra-low types (IQ2XS/IQ2S/IQ1S) cause unstable speculative-decoding acceptance rates.
5. 5.Layer-Band Protection Map — precision is assigned by layer band and tensor role (see map below), keeping critical early/mid attention & FFN paths at higher precision while late, more redundant blocks run lighter.

🗺️ Tensor Quantization Map

Architecture: 64 transformer blocks. 48 blocks are SSM blocks (tensors: ssm_alpha, ssm_beta, attn_qkv, attn_gate, ssm_out); 16 blocks are full-attention blocks (layers 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47, 51, 55, 59, 63, tensors: attn_q, attn_k, attn_v, attn_output). All 64 blocks carry ffn_down, ffn_gate, ffn_up. Norms, biases and SSM constants stay f32 in every variant (353 tensors).

SSM-block tensors (ssm_alpha, ssm_beta, attn_qkv, attn_gate, ssm_out)

FFN tensors (ffn_down, ffn_gate, ffn_up)

Full-attention tensors

Special tensors

¹ only the SSM layers inside these ranges (full-attention layers 3, 7, 11, … excluded). Exception: blk.52.ssm_alpha follows the 30–51 band (v1k: Q2K, v2k: Q3K, v3k/v4k: Q4_K).

🔧 Quantization Pipeline

Sources

• —Target: Qwen/Qwen3.8-27B official BF16 checkpoint (18 shards, 55.6 GB) → BF16 GGUF (54.6 GB, 866 tensors)
• —MTP Removal: streaming binary surgery — blk.64 (15 tensors) stripped, KV patched (block_count 65→64, nextn_predict_layers 1→0), all 851 data regions verified
• —MMProj: vision tower converted via convert_hf_to_gguf.py --mmproj, then F16
• —Draft: z-lab/Qwen3.8-27B-DFlash2-GGUF BF16 → Q2K_L (Unsloth reference distribution, 81/81 match)

Steps

1. 1.convert_hf_to_gguf.py --outtype bf16 on the official checkpoint
2. 2.MTP strip + byte-level verification against the official MTP-free reference
3. 3.Per-tensor type override files (one per variant) applied via llama-quantize --tensor-type-file
4. 4.Output verified: tensor count, per-type counts, and per-tensor types checked against the intended maps

Tools

• —llama.cpp (CUDA 13 + Vulkan build): llama-quantize, llama-gguf, llama-server
• —Custom Python scripts for GGUF binary surgery and override generation

🚀 Usage

llama.cpp Server

llama-server \
  -m Qwen3.8-27B-Q3Kmix-v2k-noMTP.gguf \
  --mmproj Qwen3.8-27B-mmproj-F16.gguf \
  -ngl 99 \
  -c 90000

Speculative Decoding with DFlash2

llama-server \
  -m Qwen3.8-27B-Q3Kmix-v2k-noMTP.gguf \
  -md Qwen3.8-27B-DFlash2-Q2K_L.gguf \
  --spec-type draft-dflash \
  --mmproj Qwen3.8-27B-mmproj-F16.gguf \
  -c 90000 \
  -fa on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl 99

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="Qwen3.8-27B-Q3Kmix-v2k-noMTP.gguf",
    n_gpu_layers=99,
    n_ctx=8192,
)

output = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello, world!"}]
)

📁 Files

SHA256 Verification

0a5b96db463dfc2f3e3861cadcfd79a272794e9215e5c8e616e83dba91efa034  Qwen3.8-27B-Q3Kmix-v1k-noMTP.gguf
14ce02bc68e823b3248fa80bdf0b35d3f436b505916592076ea3d2d6ada7bf9a  Qwen3.8-27B-Q3Kmix-v2k-noMTP.gguf
0ce2e37ac491e5d00cd9900cb4e9d31bc7f4a4575f17943d13b39c23ed409e58  Qwen3.8-27B-Q3Kmix-v3k-noMTP.gguf
2d0222974178e4ecae6ad66eb19b089672262cfb5b242fe93b547bfe5ba79f6f  Qwen3.8-27B-Q3Kmix-v4k-noMTP.gguf
10f95d16bce27608041acfece1f1c3b52974a76be8abb6e4407b2b53ccfc75b9  Qwen3.8-27B-DFlash2-Q2K_L.gguf
48853fd5605c7f8464146b14490f53d1968d3d5d9edfbdd95f8bf42263ca6312  Qwen3.8-27B-mmproj-F16.gguf

⚠️ Known Limitations

1. 1.No MTP Support — MTP head removed; not compatible with native MTP-based speculation (use the included DFlash2 draft instead).
2. 2.CPU-Only Quantization — llama-quantize has no GPU offload; builds run on CPU multithreading.
3. 3.Q2_K Floor — tensors below Q2_K intentionally excluded for speculative-decoding stability.
4. 4.MMProj VRAM — the F16 projector adds ~1–2 GB VRAM; drop --mmproj for text-only runs.

📄 License

Based on Qwen3.8-27B-UD. Quantization derivatives follow the original model license (Apache-2.0).

🔗 Related

• —llama.cpp
• —DFlash2 Draft Models
• —GGUF Format Specification

Generated: 2026-09-13 Pipeline: BF16 → GGUF → MTP-strip → pure K-mix overrides (Q2_K floor) Verified: SHA256 hashes above match the uploaded artifacts