abcdsystems/Qwen-abcd-Qwen3.8-Flash-Next-512-Q4_K_M.gguf

Qwen3.8-Flash-Next-512 — Q4KM (GGUF)

4-bit K-quant (Medium) of Qwen3.8-Flash-Next 512-expert. ~119 GB.

Why Q4_K_M: the smallest of the useful quants and — because this model's ~35 GB n-gram must stay GPU-resident — the only quant that fits a ~100 GB (5-card) rig with a usable KV pool. If your total VRAM is tight, this is your quant.

• —Base: Qwen/Qwen3.8-Flash-Next-512 (Qwen4ExpForConditionalGeneration, qwen4_exp_text)
• —48 layers (36 Gated-DeltaNet + 12 Qwen-Sparse-Attn), 512 experts / 10 active, 51.2B-param trigram n-gram at layer 2.

Load (multi-GPU)

CUDA_VISIBLE_DEVICES=<fast-cards-first> LLAMA_ATTN_ROT_DISABLE=1 \
llama-server --model Qwen3.8-Flash-Next-512-Q4_K_M.gguf \
  --split-mode layer --kv-unified --kv-unified-per-slot 16384 --parallel 8 \
  --threads <cores> --threads-batch <cores> --poll 100 --flash-attn on \
  --host 127.0.0.1 --port 8080

Required per request: "chat_template_kwargs": {"enable_thinking": false} (the model defaults to reasoning mode and will otherwise blank/delay the reply).

Do NOT pass `-ngl` or `--tensor-split` — they abort the auto-fitter. Let auto-fit place layers. The n-gram (per_layer_token_embd) stays GPU-resident by design; do not try to offload it.

Requires llama.cpp with qwen4exp support (PRs #28023 / #28123, commit 0eadefe).

Full docs, traps, and performance notes: see the base guide `Qwen3.8-Flash-Next-512-GGUF`.

Ref: ~40–46 tok/s single-stream, ~250–410 ms TTFT on 2×3090 + 3×V100.