azampatti/Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound

Qwen3.8-Flash-Next 125B-A5B INT4-AR

Qwen3.8-Flash-Next with 5 routed experts per token instead of 10, healed so it stays close to the original, quantized to int4. It runs on one DGX Spark (GB10, 128 GB) at roughly 64-70 tokens/s.

125B parameters in total, 4.8B active per token. The original activates 6B.

Everything needed to serve it is in this one repository, including the 49 GB FP8 n-gram table under ple-table/. Nothing else to download.

Run it

git clone https://github.com/azampatti/Qwen3.8-Flash-Next-Int4-FAST.git && cd Qwen3.8-Flash-Next-Int4-FAST && bash setup.sh

That builds the serving image, downloads this repository, and starts an OpenAI-compatible server on port 8000. The scripts and the full explanation are in that repo.

Serving by hand needs Saren-Arterius/qwen3.8-Flash-DGX-AutoRound, because a stock vLLM cannot serve this checkpoint's int4 + fp8 mix. Point VLLM_PLE_MMAP_DIR at ple-table/.

How it compares

Measured on one DGX Spark with our own harness, so read it as a comparison rather than a leaderboard entry. Capability is a trimmed average over 6 runs; tool use is 3 trials.

Capability was measured over 6 runs with an int8 head at 47.6, and over 3 runs with the int4 head shipped here at 46.6; the two are within noise of each other, so read it as 47 either way. The int4 head is worth about 8% more speed.

Tool use survives the cut intact. General capability gives up about four points. In exchange you get a fifth fewer active parameters and noticeably faster generation.

What was done to it

The routed experts were cut from top-10 to top-5, which costs quality because each token now sees half the expert capacity it was trained with. That loss was then healed by training only the shared expert (37.75M parameters, everything else frozen) to match what the uncut model would have produced: pure KL self-distillation, the same model at top-10 acting as its own teacher, over 14,667 examples of code, maths, general text, tool calls and long context.

Healing recovered most of the cut. Unhealed, the capability score sat at 41.8; healed, it reaches 47.6 against the original's 51.8. Text supervised fine-tuning was tried first and plateaued at 42.8, so the distillation objective is what made the difference.

What is inside

The lm_head was requantized from the original bf16 weights with AutoRound (1,000 iterations, calibration captured through this healed model). Against the bf16 head it keeps 97.2% top-1 agreement; the int8 head it replaces kept 99.6%, but it is half the size and the head is read four times per decoding step, so the trade buys real speed.

Also here: medium_chat_template.jinja and xhigh_chat_template.jinja, two optional chat templates. The default chat_template.jinja is the stock Qwen one. The medium variant stops injecting the "reasoning effort is xhigh" instruction, which cut benchmark time and tokens by about a quarter at the same score; both variants add three tool-use rules and accept <|think_off|> in the prompt.

Credits

• —Qwen for Qwen3.8-Flash-Next
• —Intel for the AutoRound int4 quantization
• —[Saren-Arterius](https://github.com/Saren-Arterius) for the hybrid checkpoint and the vLLM build that serves it on a Spark, and for the fp8 n-gram table included here

The expert cut and the healing are ours. Everything else is theirs.

README_upstream_hybrid.md and ple-table/README_upstream_table.md are the original model cards of the two checkpoints this was built from, kept as they were.