Modelpublic

vcruz305/Qwen3.8-Flash-Next-GGUF

sourceHugging Faceotherupdated 16d agoView on Hugging Face

2likes4.5kdownloads

Model Card

Qwen3.8-Flash-Next GGUF (vcruz305)

Real K-quant ladder from full HF BF16 weights with MTP heads in-main (blk.48 / nextn.*) and a shared BF16 PLE.

Layout

path	content
`Q2_K/` … `Q6_K/`	Backbone shards 00001–00006 (MTP included in tensor set)
`PLE-BF16/Qwen3.8-Flash-Next-PLE-BF16.gguf`	Shared BF16 PLE (~95.4 GiB) — use as each quant's `00007`
`imatrix/`	Importance matrix used for Q2–Q6
`scripts/link-ple.sh`	`ln`/`cp` PLE → `…-00007-of-00007.gguf` per quant
`MTP/`	Optional standalone MTP draft heads (if present)

Setup

bash

bash scripts/link-ple.sh

Approximate backbone sizes (GiB) + shared PLE

quant	backbone	MTP tensors
Q2_K	~49.5	≥20
Q3KM	~62.5	≥20
Q4KM	~80.2	≥20
Q5KM	~91.3	≥20
Q6_K	~107.8	≥20
PLE BF16 (shared)	~95.4	—

Run (DGX Spark / unified memory)

Requires qwen4exp-capable llama.cpp (PR #27742 class).

bash

llama-cli \
  -m Q4_K_M/Qwen3.8-Flash-Next-Q4_K_M-00001-of-00007.gguf \
  --load-mode mmap \
  -ngl 99 \
  -ot "per_layer_token_embd.weight=CPU" \
  -c 1024 -n 64 -st --temp 0 \
  -p "The capital of France is"

Do not mlock the PLE on 128G unified-memory boxes. PLE stays NVMe-backed via mmap.

MTP speculative decode

Weights include MTP (nextn_predict_layers / blk.48). Runtime --spec-type draft-mtp needs a llama.cpp build with qwen4exp graph_mtp (not all mainline builds yet).

Build notes

—Converter: llama.cpp qwen4exp with MTP export enabled
—Imatrix: AtomicChat-compatible matrix
—PLE left BF16 (not re-quantized); tok emb Q80; output Q6K