todiadiyatmo/Qwen3.8-Flash-Next-W4A16-Attn8-FP8PLE

Qwen3.8-Flash-Next-W4A16-Attn8-FP8PLE

A lightly requantized derivative of Intel/Qwen3.8-Flash-Next-W4A16-AutoRound. The routed experts keep Intel's original 4-bit AutoRound weights (W4A16); a selected set of non-expert projections (the QSA attention and GDN linear-attention matrices, 156 in total) is converted from BF16 to symmetric per-channel 8-bit (Attn8), and the PLE lookup table is replaced with an FP8 version (FP8PLE). Since 13 Sep 2026 the checkpoint also carries a 4-bit/8-bit MTP draft head (1.33 GiB, from klee100) in place of the original BF16 MTP head (4.86 GiB), so MTP speculative decoding can be enabled in vLLM.

The Attn8 layout was chosen deliberately for the AllSpark W8A16 kernel (Alibaba's DashInfer GEMM) on Ampere-class GPUs such as the RTX 3090. vLLM removed AllSpark in #58001 (22 Sep 2026); newer builds (recipe v1.4.0 and later) serve the same INT8 weights with Marlin W8A16. Both read half the weight bytes of BF16 at decode time.

Benefits on our four-RTX-3090 setup:

• —Attn8: reduces the size of the selected attention/GDN weights and helps decode speed. Reference runs reached about 72 tok/s, compared with about 53 tok/s for Intel (see the benchmark conditions below).
• —FP8PLE: reduces the PLE lookup table's host RAM footprint from about 95 GiB to 48 GiB. This is the table alone; the server needs additional RAM.
• —MTP draft head (4-bit/8-bit): enables speculative decoding with 2 draft tokens. On the reference setup decode goes from about 84 to 130 tok/s (prose) and 84 to 161 tok/s (code); the KV pool shrinks by about 45%. Prefill is unchanged.
• —TP2 × PP2 recipe: provides roughly 995,000 tokens of shared KV cache capacity without MTP (about 540,000 with MTP), about twice the TP4 layout. The model has only two KV heads, so TP4 duplicates heads across GPUs. TP2 × PP2 instead uses two GPU pairs, each handling half the layers, halving attention KV memory per token per GPU. This benefit comes from the serving layout, not FP8PLE. The shared cache pool is not the per-request context limit of 262,144 tokens.

Run with vLLM

Use the four-RTX-3090 vLLM recipe for setup instructions, the Docker image build, and the required patches. This checkpoint was tested with that patched build, not stock vLLM. Use the recipe's explicit Sharp chat-template setting.

MTP is off by default. To enable it, add --speculative-config={"method":"mtp","num_speculative_tokens":2} (the recipe ships this as recipe/yml-tp2-pp-2-mtp.yml). It needs recipe v1.1.0-eed1f3d0 or later: older vLLM builds cannot run MTP under pipeline parallelism and do not know this model's quantized draft head.

Quantization Method

Runtime kernels (how the INT8 weights are actually served)

Current vLLM (recipe v1.4.0, nightly `29468dde` and later): AllSpark no longer exists in vLLM (#58001), so the 156 INT8 matrices are served by Marlin W8A16 (Using MarlinLinearKernel for AutoGPTQLinearMethod). The files on disk are unchanged. With recipe v1.4.0 the checkpoint was verified to boot and pass its functional probes (tool calls, reasoning, prefix cache, vision); the performance tables in this card were measured with AllSpark and have not been re-measured on Marlin.

• —MTP draft head: loaded by vLLM's qwen4_exp MTP module through the same INC config; its INT4 experts and INT8 projections both run on Marlin, like the body. The draft head lives on the last pipeline stage. DFlash and DSpark are not available for this model.

Historical (recipe up to v1.3.0, nightly `eed1f3d0` and earlier), AllSpark:

The INT8 layout above was chosen to land on a specific kernel. On Ampere GPUs, vLLM routes the 156 INT8 matrices to AllSpark, the W8A16 GEMM from Alibaba's DashInfer engine that was contributed upstream and lived in the vLLM tree until #58001 (csrc/quantization/gptq_allspark/, wrapper AllSparkLinearKernel). The boot log confirms it for every worker:

INFO [auto_gptq.py] Using AllSparkLinearKernel for AutoGPTQLinearMethod

• —Why AllSpark: on Ampere it accepts INT8 only as symmetric per-channel weights (group_size = -1, no zero points) with per-GPU matrix dimensions divisible by 16, matching the Attn8 layout. Grouped INT8 (e.g. g128) uses a different kernel path, such as Marlin, subject to its own shape requirements. Small GDN control projections and shared experts remain BF16.
• —What it does: weights are repacked once at load without changing the files on disk. For small batches, including decode, it reads the INT8 weights and converts them inside the compute kernel. For larger batches, it converts the weights to BF16 and uses cuBLAS. In this build the switch is at 1,024 input rows, not the total prompt length.
• —Measured trade-off: an earlier controlled TP4 experiment measured cold prefill about 3–5% slower than the Intel checkpoint with BF16 attention/GDN projections. This is a result for that setup, not a fixed penalty for every recipe. Intel's routed experts are INT4 in both cases.

Memory comparison

These are reference measurements, not a controlled memory comparison. The profiles differ as described below; the loaded-weight difference should not be read as an isolated measure of the INT8 savings. The PLE column covers only the lookup table, not total server RAM.

KV cache, prefill, and decode comparison

Benchmark conditions: these are reference runs, not a controlled A/B comparison. The Intel run (2 September 2026) used an older nightly, a text-only profile and GPU memory utilization of 0.95. The later runs used vision enabled and utilization of 0.94; prefill depth was 90K for the e962733e row and 100K for the eed1f3d0 rows. All used TP2 × PP2. MTP draft acceptance in the n=2 run: 70% during the benchmark (position 1: 84%, position 2: 68%; mean accepted length 2.4 of 3 tokens per step). The no-speculation KV pool is given as a range from two boots of the same configuration.

The KV pool is shared across the whole engine and varies by about ±2% between starts. Decode was measured with one prose request at a time; differences below about 11% are within this rig's observed noise. Prefill measures processing without prefix-cache reuse. All reference speeds used a P2P-patched driver; benchmark again on other setups.

Behavioural test (club-3090 quality-test.sh, 75 scenarios, pass@1)

Three fresh server starts per model, alternating between Intel AutoRound (base) and W4A16-Attn8-FP8PLE, using the same engine and configuration. Each start runs the streaming tool-call probe (60 tests, --thinking on --tool-choice both --repeat 3) and the 75-scenario quality suite once before the server is stopped. The operating system's file cache is not cleared.

Quality tests used short prompts, one request at a time, temperature=0, max_tokens=4000, and GPU memory utilization of 0.88 for both models, rather than the recipe's 0.94. The suite uses club-3090's quality-test.sh --medium, scored by benchlocal-cli. These settings differ from the standard benchmark defaults, so compare scores only with runs using the same protocol.

Both models have the same mean (67.7/75) and range (66–69), and both passed all streaming tool-call tests on every start. No test pack differs by more than one scenario on average.

No quality difference was detected in this suite. This does not establish identical quality on every task or at long context lengths. A single run is not enough to rank models with small score differences.

Raw per-boot scores (pass@1, scenarios passed out of 15 per pack; streaming probe cells passed out of 60):

MTP profile, 13 Sep 2026 (vLLM `eed1f3d0`, recipe `v1.1.0-eed1f3d0`, one boot, same protocol): 68/75 pass@1 (69 pass@3), streaming tool-call probe 60/60. The same two scenarios that fail without MTP (TC-11, DE-10) fail here too. Speculative decoding with greedy verification does not change which tokens are produced; this run confirms the quantized draft head does not break tool calling or structured output.

Still untested: hours-long stability, sustained traffic with multiple simultaneous requests, every request type at the full 262K context limit, and performance without P2P or on GPUs other than the RTX 3090.

Acknowledgements

Changelog

• —26 Sep 2026 — documentation only. vLLM removed the AllSpark INT8 kernel (#58001); from recipe v1.4.0-29468dde the INT8 projections run on Marlin W8A16. Weights unchanged. Verified to boot and pass functional probes on v1.4.0; the performance numbers in this card were measured with AllSpark and have not been re-measured.
• —13 Sep 2026 — 4-bit MTP draft head. model_extra_tensors.safetensors (BF16 MTP head, 4.86 GiB) replaced by mtp-model-00001-of-00002.safetensors and mtp-model-00002-of-00002.safetensors (1.33 GiB, from klee100). config.json gained quantization rules for mtp.* modules (the former rule that forced the MTP head to BF16 was removed); model.safetensors.index.json, quantization_config.json and MANIFEST.json updated. Body weights unchanged. Tested with recipe v1.1.0-eed1f3d0 (vLLM nightly eed1f3d0), TP2 × PP2, MTP n=2: decode 132 / 160 tok/s (prose / code), KV pool 527K, quality 68/75, tool calls 60/60. The previous revision remains available with --revision 12c006bb0496db523dce34ca9105ee5d84a01a31.
• —8 Sep 2026 — first release. W4A16 experts (Intel), INT8 attention/GDN projections, FP8 PLE table, BF16 MTP head. Tested with recipe v1.0.0-e962733e (vLLM nightly e962733e).