Modelpublic

Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-MTP_int4RTN

sourceHugging Faceotherupdated 18d agoView on Hugging Face

0likes281downloads

Model Card

Qwen3.8-Flash-Next W4A16 AutoRound hybrid — MTP draft experts in int4 (RTN)

Variant of Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid for [Saren-Arterius/qwen3.8-Flash-DGX-AutoRound](https://github.com/Saren-Arterius/qwen3.8-Flash-DGX-AutoRound) (Qwen3.8-Flash-Next on a single DGX Spark / GB10 via vLLM).

Only one thing differs: the MTP draft layer's 512 routed experts, which Intel's AutoRound release leaves in bf16 (~4.7 GiB, excluded via -:.*layers\.48\..*), are quantized to int4 GPTQ g128, symmetric, RTN — the same recipe and the exact tensor layout Intel uses for the 48 main layers' experts — and that exclusion is dropped from quantization_config, so vLLM serves the drafter on GPTQ-Marlin. All 67 weight shards are byte-identical to the hybrid repo; model_extra_tensors.safetensors (5.0 GB → 1.49 GB), model.safetensors.index.json and config.json are the changed files. Built by the repo's tools/quantize_mtp_experts_int4.py.

Why: the drafter is on the decode hot path (MTP=3 runs it three times per step).

Measured on a DGX Spark (bench_qwen35.sh, temperature 0, MTP=3, second run), against the hybrid repo:

	bf16 draft (hybrid repo)	this repo	this repo + `DRAFT_VOCAB=1`
Code / JSON / LongCode tok/s	57.3 / 61.3 / 57.9	59.1 / 62.6 / 60.3	62.0 / 66.0 / 62.4
draft acceptance (of 3)	88.0% (2.64)	88.5% (2.65)	87.3% (2.62)
weights resident	~71.4 GiB	67.9 GiB	67.9 GiB

Output quality is unaffected by construction — the target model verifies every drafted token; only the draft's cost and acceptance can move, and acceptance did not.

Idea: upstream blazux/qwen3.8-Flash-DGX's hybrid-mtp mode (PR #11 by @pfy, after thavoc's graft write-up) grafts NVFP4 draft experts onto their hybrid checkpoint; this does the equivalent with RTN int4 so it needs no new kernel path next to GPTQ.

Use

bash

git clone https://github.com/Saren-Arterius/qwen3.8-Flash-DGX-AutoRound
hf download Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-MTP_int4RTN --local-dir /models/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-MTP_int4RTN
hf download Saren/Qwen3.8-Flash-Next-ple-table-fp8 --local-dir /models/ple-table-fp8
# then: docker build + serve.sh with MODEL_DIR pointed here — see the GitHub README

Weights carry Qwen's license (MAU/revenue clause) — review before production use.