halt95/Qwen3.8-Flash-Next-W4A16-Merlin

Qwen3.8-Flash-Next-W4A16-Merlin

Three full 262K sessions on four RTX 3090s. Qwen3.8-Flash-Next with an 806,792-token KV pool, MTP speculative decoding and vision on consumer Ampere (sm_86, 24 GB each, PCIe, no NVLink bridge), with thinking-on decode at parity with the single-session TP4 build and prefill 10–26 % faster.

v2.0.1 (2026-09-18): the same weights serve with an 806,792-token KV pool, three full-context sessions resident at once on the four cards, through the v2.0.1 engine in the companion repository halt95/qwen38-flash-next-3090s: the v2 shape (2026-09-17) plus one engine fix, structured output under concurrency (What changed in v2.0.1). v2 changes the serving shape, not the checkpoint: tensor parallel 2 × pipeline parallel 2 with expert parallel, the FP8 n-gram table served to the GPUs over a host-mapped, fail-closed pull transport, and the token-embedding tables of both the target model and the MTP drafter moved out of VRAM into pinned host memory, read by a device-mapped lookup inside the captured cudagraph. Thinking-on single-stream decode is at parity with the v1 TP4 build (thinking-off at 4K runs 5 % below it, documented) and prefill is 10–26 % faster. The v1 build (TP4, 342,912-token pool) still serves this checkpoint and stays in the repository.

This is an Ampere build. Stock vLLM cannot serve Flash-Next with an FP8 KV cache on sm_86 (Triton there has no fp8e4nv type, so the FP8 read path is a custom integer-decode kernel), and it cannot fit the model's n-gram embedding table on 24 GB cards without PLE offload, which is still an open upstream PR (#53899). That table is 102 GB in Intel's BF16 release and ~48 GiB in the FP8 form used here. The checkpoint therefore runs on a patched vLLM that adds both, published with pinned build scripts in the companion repository. If your cards have native FP8 and enough memory for the BF16 table, you likely do not need this checkpoint; Intel's release is the natural starting point.

What is in the checkpoint

The shard files are numbered with gaps (model-00002-of-00017 and model-00016-of-00017 do not exist); the index references only the 27 weight files that do, so a download that lists 25 model-* shards plus model_extra_tensors and mtp-routed-experts-int4 is complete.

The v2 delta is one key in `config.json`, inside the `text_config` object: "ple_embedding_dtype": "float8_e4m3fn", which is how the v2 engine selects the FP8 n-gram path (the v1 engine used the VLLM_PLE_FP8_GLOBAL_SCALE=1 environment variable and reads the key too). Every weight file is byte-identical to the 2026-09-08 upload. If your copy of config.json predates the key, scripts/make-e1-config.py in the companion repository inserts it (in text_config; a top-level copy of the key is ignored by the engine) without touching anything else. This repository's config.json carries the key since the v2 refresh.

No VnimanieAI tensor is in this checkpoint; that lineage is the previous checkpoint it replaced and the baseline in the quality table below.

Attention is left in BF16 on purpose: only 12 of the main model's layers are true attention, every token passes through all of them, and their K/V projections feed a cache that is already FP8. Packing them to INT4 would save about 0.37 GiB per card; whether it costs quality was not tested, so it was not done.

What changed in v2.0.1

Structured output now works under concurrency. In v2 a response_format request could fail with HTTP 500 while other requests were decoding: the draft hand-off kept one unidentified slot that the alternating pipeline-parallel microbatch overwrote, the -1 placeholders left behind gave the grammar bitmask an all-allowed row, and the unconstrained token then failed the state machine. v2.0.1 ports two upstream pull requests, both open upstream at the time of writing: PR #54442 (never leave an unmasked row for a draft slot the scheduler did not schedule) and PR #56802 (key draft snapshots by scheduler step, so a request is verified against the drafts its own step consumed).

Measured with the shipped serve command, 80 structured requests over four load cells, 10 idle and 10 under load each:

Idle was 10/10 in every cell on both; every v2.0.1 response body validated against its schema, and the run logged no grammar rejection and no terminated request. No throughput regression was found: on the reference host's two-boot ladder across 4K, 32K, 131K and 261K in both thinking modes v2.0.1 lands between 0.978 and 1.028 of the v2 median, median inter-token latency within 0.21 ms, tokens per step unchanged. The prefix-cache equivalence block fails identically on v2 and v2.0.1 (open behaviour, below), unchanged by the fix. The checkpoint, the serve command and the KV budget are untouched — this is an engine fix, and the weights are byte-identical. These figures are maintainer-reported; their record is not published, and the two upstream pull requests are the independently readable evidence.

Measured

Hardware for everything below: 4× RTX 3090 at 220 W, Gen4 x16 to every card, peer-to-peer over the open-kernel-module P2P patch, an EPYC 7532 with 192 GB ECC of which the serving container is allocated 96 GB. Three runs are reported, and they are not the same configuration.

Serving: the v2 gate (2026-09-17), TP2 × PP2 + EP, MTP K=3

Numbers from the pre-registered two-arm gate that qualified the v2 tag: five boots of v2 alternating with five boots of the v1 TP4 build on the same box, one frozen manifest, three streamed sends per cell, T=0 with a fixed seed, judged as medians (benchmarks/2026-09-17/BENCH-CARD.md in the companion repository, every boot's numbers in it).

† v1 at 261K was measured on 2026-09-18, after the gate, in one boot with three sends (170.1 / 124.6 tok/s; median event interval 19.37 / 19.40 ms; 3.24 / 2.39 tokens per step, thinking on / off). Every other v1 cell is a median of five boots; the 261K depth was preregistered for the v2 arm only, so v1 prefill and time to first token at that depth are not measured.

Decode is the streamed event rate over the answer, client-timed. With MTP the honest pair is the median event interval (v2 18.7 / 18.7 / 19.1 ms against v1 18.6 / 18.7 / 19.2 at 4K / 32K / 131K) and tokens per step, both in the card. One cell of the gate is recorded as a shortfall: thinking-off decode at 4K reads 0.92–0.99 of the v1 build on every one of the five boots, four of them below the gate's 0.97 rule, at an identical event interval (fewer accepted draft tokens per step, clustered per boot; cause not established). The served entry defaults to T=1.0 / top-p 0.95 / top-k 20; the gate numbers were not re-measured under those defaults.

Serving: the v1 card (2026-09-08), TP4 + EP, MTP K=3

Captures [1,4,8], the shipped sidecar, through the front door, thinking on, an image in every depth prompt (benchmarks/2026-09-08/BENCH-CARD.md; 2 clip-counter lines over the run, zero errors): single-stream decode 166 / 174 / 172 / 163 tok/s at 4K / 32K / 131K / 260K prompt tokens, time to first token 0.9 / 7.0 / 32.0 / 75.7 s at the same depths, deepest measured prompt 260,566 tokens, MTP acceptance 2.43 in the card's logging window (2.4–2.9 across served logging windows, maintainer-reported). End to end, a cold 260K prompt is dominated by its prefill: about 76 s on the v1 build and 51 s on v2 (the table above).

Quality: paired per question (2026-09-03), v1 build, MTP K=2

A different instrument from the gate above, on a different serving shape (captures [1,3,6]). The baseline is the checkpoint this one replaced: the same MTP and GDN packs applied to the VnimanieAI W4A16 checkpoint, whose attention projections are INT4 where Intel's are BF16.

Read those honestly. None of the three task deltas reaches significance at these sizes. An earlier comparison (2026-09-02, the unpacked Intel graft at a different serving shape, so not this exact checkpoint) had the same direction and was significant on GSM8K and ARC; the two runs share a baseline lineage and are not independent replications. The verbosity result is one run of 40 questions at xhigh (accuracy 32/40 vs 31/40); at the served low effort verbosity was unchanged (median 1,054 vs 1,059) and accuracy read 32/40 vs 35/40. In that K=2 gate this checkpoint decoded about 5 % slower than the previous one at short context and equal at depth. The v2 gate's quality row (divergence from the BF16 teacher on 24 held-out prompts, above) is a different instrument and is not comparable to this table. An earlier single-capture quality screen of the v2 campaign is recorded as a FAIL and stays cited as such in the companion repository; the two-arm gate above replaced it as the instrument.

Known behaviours of the Qwen3.8-Flash-Next architecture in vLLM

These are behaviours of the architecture as implemented upstream in vLLM (the hybrid Gated-DeltaNet / sparse-attention / MTP execution path and its hybrid KV manager). They are not introduced by this checkpoint's quantisation and not by the v2 engine's patches; each is documented in the companion repository, and none affects the correctness of answers in the gate. Read them before serving.

• —An occasional empty completion on a warm repeat of a long cached prefix (upstream class, vllm-project/vllm #53912; seen on the v1 TP4 build too): HTTP 200, finish_reason: "stop", zero tokens. Per boot, not per request; none in the 10 boots of the v2 gate, three in 23 earlier gate boots. Mitigation shipped in the serving recipe: clients retry once; it returned the correct answer in every observed case (a mitigation, not a guarantee).
• —Prefix-cache blocks of a session that finishes while other long sessions are still decoding are dropped (upstream hybrid KV manager path). Cost: first-token latency on that session's next turn; the answer is unaffected. Prefix-cache hits land in 3,200-token aligned blocks, so prompts under about 6.4K tokens cannot hit.
• —A rare illegal-address engine death (upstream sparse-attention code path; two occurrences over the whole v2 campaign). Bounds guards in the v2 engine turn a recurrence into a named, counted failure when their reader is on; the reader is off in the served profile. Not fixed; no rate is claimed.
• —Greedy T=0 is not byte-reproducible with this architecture (bf16 near-ties resolved differently by the sparse indexer's top-k and the expert permutation); documented, accepted.
• —Thinking-off decode at 4K runs below the TP4 lane, 0.92-0.99 of it on every one of the five gate boots, four of them under the 0.97 rule, at an identical event interval (fewer accepted draft tokens per step). It is the one cell that makes the gate's machine verdict FAIL; the tag shipped on an explicit decision to document it rather than block on it. See "Measured" above.

The KV-scale sidecar

qsa_kv_scales_262k.json holds static FP8 scales for the 12 attention layers of the main model. It is the same file for v1 and v2 (byte-identical). It was calibrated on this checkpoint on 2026-09-08: eager (no CUDA graphs), no speculative decoding, text at 1,971 / 7,989 / 31,975 / 127,805 / 199,888 / 254,875 prompt tokens plus image prompts at 2,848 and 8,841 tokens, merged as the maximum over the four TP ranks with a 10 % margin. The rank dumps and the merge script ship with the companion repository and reproduce this file byte for byte.

Gated at the v1 served shape (MTP K=3, full CUDA graphs, 262K, prefix caching on), each arm a fresh server: daily lane check 4/4 (text, one image, two images, exact needle at 207,495 tokens); 24 clip-counter increments on 3 layers during the gate, against 496 on 9 layers for the earlier sidecar calibrated on the previous checkpoint. Served by a build made from the public repository with the final v1 patch revision (strict loader, float32 clip ceilings), the same file read 0 increments and 4/4. Without a sidecar the FP8 path runs at scale 1.0. The draft head's own attention layer is not in the sidecar and runs at scale 1.0. The v2 gate ran 10 boots with the clip counter compiled in and off; the sidecar's effect on v2 decode was not measured separately.

Serving

v2.0.1 (recommended: 806K pool, three 262K sessions)

From the companion repository at release v2.0.1. That release pins vLLM fork tag v2.0.1 = commit ad5c3c223999de577b04cdb9caeab2dcb76b61b9, tree 5426cf19586a66946a1d8f7e68f48bdc7f52d62d (upstream/PIN-v2); scripts/build-v2.sh asserts both the commit and the tree after checkout, so a build that does not match the pin fails rather than serving something else.

1. 1.Build with scripts/build-v2.sh: the pinned upstream vLLM commit plus the community Flash-Next commits and the v2 delta (shipped as a git bundle release asset, verified by commit and tree hash), a fresh Python 3.13 venv from 199 pinned packages, compiled ops from the upstream precompiled cu130 wheel pinned by sha256 (no C++ change in v2, so no compiler needed to serve).
2. 2.Add the config key if your download predates it: python scripts/make-e1-config.py /path/to/Qwen3.8-Flash-Next-W4A16-Merlin.
3. 3.Serve with scripts/serve-v2.sh /path/to/Qwen3.8-Flash-Next-W4A16-Merlin. It is the served entry with every variable exported; the shape it runs is TP2 × PP2 + EP (VLLM_PP_LAYER_PARTITION=25,23), MTP K=3 probabilistic, --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 1024, --kv-cache-memory 4100000000 --kv-cache-dtype fp8_e4m3 with the sidecar, FULLANDPIECEWISE cudagraphs with captures to 32, prefix caching, the PLE pull transport (VLLM_PLE_CPU_OFFLOAD=1, home GPU 0) and the host-resident embeddings (VLLM_HOST_EMBED_TABLE=1). The exact command and environment are in the script and in the bench card.

Settings that are load-bearing on v2:

• —Host RAM: 96 GiB (98,304 MiB) is the qualified allocation. The ~48 GiB FP8 n-gram table and about 4.2 GiB of pinned embedding tables (1,064 MiB per rank) live there; the measured resident floor is about 69 GiB, the rest being reclaimable page cache, and the boot peak is not measured. /dev/shm needs at least 1 GB (a Bus error at boot means it is undersized).
• —--kv-cache-memory 4100000000 is the highest pin that boots with a complete clean row set on 24 GB cards; the non-KV headroom that remains is about 1 GB per card. The 25/23 layer partition is part of that budget.
• —--max-num-batched-tokens stays at 1024 on this shape. The deep-prefill transients that engine-killed 2048 and 4096 on earlier configurations sit outside the torch pool.
• —Capture sizes cover max-num-seqs × (K+1) = 32.
• —VLLM_HOST_EMBED_TABLE=1 is on by default and is a torch compile-cache factor; turning it off costs the 608 MiB per card back and needs its own VLLM_CACHE_ROOT.
• —Do not set PYTORCH_CUDA_ALLOC_CONF=expandable_segments (v1 measured it at −27 % four-stream aggregate).
• —Clients retry once on an empty completion (above).
• —CUDACachingAllocator ... OOM warning lines can appear when several long sessions grow at once: the upstream sparse-attention indexer's prefill logits buffer (1 KiB per token of context, 268 MB at 262K) asked for a contiguous block, the allocator flushed its cache and the retry succeeded. Warnings, not failures (maintainer-observed: 22 lines, 0 failed requests in a three-session run); a failing retry would mean the ~1 GB non-KV headroom per card is exhausted.

v1 (TP4, 342,912-token pool)

scripts/build.sh and scripts/serve.sh in the same repository: vLLM v0.28.0, the eight community Flash-Next commits, six patches, the compiled-ops wheel pinned by sha256 and the sm86 GDN decode kernel rebuilt (needs CUDA 13 and gcc ≥ 11.3). Serve shape: TP4 + EP, MTP K=3, captures `[1,4,8]`, `--max-num-seqs 2`, `--max-num-batched-tokens 2048`, `--kv-cache-memory 2600000000` as shipped in `serve.sh` (2.9e9 = pool 342,912 tokens on the reference host since 2026-09-10; the v2 gate's v1 arm ran at 2.9e9), `VLLMPLEFP8GLOBALSCALE=1`, `VLLMPLEOFFLOADHOME_DEVICE=3. The 2026-09-08 card's load-bearing notes still apply to it: the capture list must cover max-num-seqs × (K+1) (a [1,3]` list once collapsed two-stream throughput to 18 tok/s per stream), and 4096 batched tokens peaked at 24.1 of 24 GiB at 262K.

Limitations

• —Requires the patched vLLM above (the v2.0.1 tag or the v1 build). Upstream vLLM merged Flash-Next model support on 2026-08-31 (#53896) but not PLE offload (#53899); whether a stock build loads this checkpoint's compressed-tensors config was not tested.
• —The v2 evidence boundary: the gate is a maintainer measurement on the reference host with a frozen, hash-pinned manifest; the bench card is published in the companion repository, the underlying close records, raw streams and boot logs are not. The v2.0.1 requalification figures (in What changed in v2.0.1) are maintainer-reported and their record is not published; the two upstream pull requests the fix ports are public and can be read independently. What you can reproduce independently: the source tree (commit and tree hash), the environment, the sidecar and the serve command. The v2 build was reproduced from the bundle on the reference host; no third-party reproduction is documented.
• —Quality gains over the previous checkpoint are directionally consistent but individually non-significant; the clearer effect is verbosity at high reasoning effort, from one 40-question run.
• —The five known behaviours above are documented, not fixed; the empty warm completion and the prefix-cache loss remain open; structured output under concurrency is fixed in v2.0.1.
• —Evaluated on one hardware configuration (4× RTX 3090, 220 W, Gen4 x16, P2P enabled). Peer-to-peer on your driver is assumed, not validated.
• —Attention, norms, embeddings and lm_head are BF16, so this is not a maximally compressed checkpoint.

Lineage and credit

• —Qwen for Qwen3.8-Flash-Next and the licence (Qwen Community License 1.0, vendored here as LICENSE; its notice applies to these weights as a derivative work).
• —Intel for the AutoRound expert quantisation; the maintainer's reading is that the expert rounding is where the measured quality difference comes from, though attention format also differs between the two compared checkpoints.
• —RadixArk for the FP8 PLE table.
• —DominikBucko for the MTP INT4 packing recipe this one adapts.
• —The community PLE-offload patch stack (peakcrosser7, vLLM PRs #53896 / #53899) that makes Flash-Next serveable on Ampere, and the upstream fixes the v2 engine carries ahead of its base (credited in the companion repository).
• —The v2 host-mapped pull transport and the host-resident target and drafter embeddings inside the cudagraph are the maintainer's own work.