TheDrainFlorist/Qwen3.8-Flash-Next-VQ-2.1bpw

Qwen3.8-Flash-Next-VQ-2.1bpw

45.8 GiB — a 335 GiB frontier MoE on 64 GB machines.

v2 (2026-09-15): per-layer mixed codebooks, chosen by measurement. Beats the v1 build on all three referee corpora at 0.025 GiB SMALLER. See "What changed in v2".

A data-free vector-quantized build of Qwen3.8-Flash-Next (180B total / 10-of-512 active, 51.2B n-gram PLE, vision) for Apple Silicon. Stock mlx-lm, no patches — the VQ runtime ships inside the checkpoint as model.py. Built with VQLab.

MoE experts at a per-layer mixed geometry: d=8/K=16384 for most expert gate/up, d=4/K=256 on the nine layers measurement said pay for it (L27–L33, L35, L47), and d=4/K=256 for every expert down_proj — which also removes the padded-tail waste the v1 build carried. PLE n-gram tables at d=8/K=256 (8-bit rows); layers 0–1 keep their d=2/K=256 front protection.

Size convention on this card: 45.755 GiB is the trunk as it exists on disk. The MTP sidecar (mtp-head-q6.safetensors) is a further 2.140 GiB and is only loaded if you enable it.

The affine builds compared against below are our own conversions made with the same tooling, scored on the same instrument.

[image]

Requirements — read this first

This checkpoint ships its own runtime inside model.py; config.json points to it with model_file. Your loader must honor that key or nothing works.

If you see Received N parameters not in model: with keys like language_model.model.model.…, your mlx-vlm is too old and never read the bundle — it fails for text too. pip install -U mlx-vlm. This is not a bug in the checkpoint.

Requirements

This model needs an `mlx-lm` that has the `qwen4_exp` architecture, which no released version has yet. The architecture is in ml-explore/mlx-lm PR #1788, still unmerged as of 2026-09-09. On a stock pip install mlx-lm you will get:

ModuleNotFoundError: No module named 'mlx_lm.models.qwen4_exp'

That is the architecture missing, not a problem with this artifact. Until the PR merges:

pip install git+https://github.com/ml-explore/mlx-lm.git@refs/pull/1788/head

python -m mlx_lm generate \
  --model TheDrainFlorist/Qwen3.8-Flash-Next-VQ-2.1bpw \
  --prompt "Explain vector quantization briefly." \
  --max-tokens 512

The VQ runtime itself needs no patches — it ships inside the checkpoint as model.py and stock mlx-lm executes it. The PR is required only for the base architecture.

Distributed serving via [exo](https://github.com/exo-explore/exo): upstream exo pins a released mlx-lm that lacks qwen4_exp, so stock exo cannot serve this model. Our exo fork, branch `mtp-stage1` serves it, and speculative decoding there is one opt-in knob: set EXO_MTP=1 in the worker's environment and it loads mtp-head-q6.safetensors and drafts; leave it unset (the default) and the sidecar is never read — no memory cost. Budget ≈2.2 GiB extra resident when drafting is enabled. On a 64 GB machine the trunk alone is 45.8 GiB, so the head fits but leaves little headroom for anything else. Measured on this rung through exo (M4, single node): 23.7–25.5 tok/s decode at acceptance 0.82–0.88.

Changelog

2026-09-19 — vision fix

Images did not work on any revision before this date: loading with an image failed with AttributeError: module 'custom_model' has no attribute 'TextConfig', because the bundle bound a text-only architecture. Text was never affected. Only model.py changed — weights and config.json are unchanged, so no re-download. Verified: text via mlx-lm and an image through the model via mlx-vlm 0.6.17.

2026-09-09 — runtime refresh

Runtime refresh. The bundled model.py is updated so downloaders run exactly the code that was benchmarked; dense bundles now carry both runtimes.

• —Faster prefill on affected geometries, measured per rung: a device-codebook kernel arm for large-codebook geometries (up to 1.46x on affected rungs), a ragged-subvector relaxation (up to 1.34x on affected rungs), a fused d8 arm, and a routing memo (≈2%). No blanket speedup is claimed across the lineup — gains apply only where the geometry engages the new paths.
• —Output quality is unchanged: the kernel changes are bit-identical or 1-ULP-equivalent, and the routing memo is bit-identical (logits checksum verified).
• —Speculative decoding (MTP): on repos that ship mtp-head-q6.safetensors, the sidecar works with the exo fork branch mtp-stage1 (github.com/noahzelezny/exo) — launch each node with exo --mtp (or set EXO_MTP=1). Note: with MTP enabled, exo serves requests sequentially (the batch engine has no MTP path), so leave it off for concurrent / multi-agent workloads.

2026-09 — bundle refresh

Bundle refresh.

model.py is updated:

• —Faster MoE expert kernels (threadgroup-occupancy fix, device-direct activation reads, template-free dispatch, and a simdsum reduction). Measured on this family: decode 17.4 -> 18.8 tok/s (+8%, A-B-A, M3 Ultra). All kernels verified bit-identical to the previous bundle except the reduction, which is 1-ULP equivalent with a measured quality delta of exactly zero (identical perplexity to 16 digits, KL 0.0 between old and new logits). `VQD8_SS=0` restores the bit-identical legacy reduction.
• —The runtime caps MLX's buffer-reuse cache by default (VQLAB_CACHE_LIMIT_GB overrides, 0 disables), so long-prompt prefill no longer balloons transient memory — peak stays near resident instead of a multiple of it.
• —The bundle loads under both mlx-lm and mlx_vlm.
• —Nothing you have downloaded breaks: weights unchanged, only model.py and config.json keys replaced; prior revisions stay pinnable by commit hash.

Peak memory, measured externally: process RSS sampled at 5 Hz from outside across load + 2048-token prefill + 128-token decode peaks at 45.8 GiB — the weight size; the runtime returns transient prefill allocations as they free, so peak ≈ resident (without the MTP sidecar; add ≈2.2 GiB when enabled).

Measured results

Prose referee, 2048 tokens. KL against the bf16 teacher's cached top-64 (captured mass 0.9626 for every row — same cache, same positions). Sizes include the 333-tensor bf16 vision tower (0.84 GiB). Every row re-measured 2026-09-15 on one corrected scorer; earlier published figures for this family came from a streamed scorer since found to disagree with a direct full-model forward, and are not comparable to these.

Rank by KL, not perplexity. KL and top-1 order this ladder as the bit budgets predict; perplexity does not. The 3.2bpw rung reads 5.1684 — lower than 4.4bpw, 5.5bpw and the 178 GiB q8 — while sitting at more than five times q8's divergence from the teacher. Perplexity is a mean over finite text and absorbs offsetting errors; KL measures the distribution itself. (The bf16 teacher's own ppl is omitted: it is 0 KL by definition, and a 335 GiB scoring pass does not complete on this hardware.)

Every VQ rung beats the affine rung at or above its size. VQ-2.1bpw at 45.8 GiB is 3x closer to the teacher than affine q3 at 75 GiB, and level with affine q4 at 96 GiB (340 vs 307 KL, 80.2% vs 80.0% top-1) at less than half its size. VQ-4.4bpw beats affine q5 outright while being 20 GiB smaller.

What changed in v2

Allocation was measured, not chosen. vqlab alloc-sweep built and scored 18 whole artifacts to trace two curves — what demotion saves and what promotion buys — and the result contradicted two reasonable guesses:

• —No layer was cheap enough to demote. Every arm that downgraded cold layers lost on all three corpora.
• —The last promotion was decided by WHICH layer, not how many. The ranking's 9th choice lost on all three corpora; its 10th (L47, in the tail) won on all three at identical bytes. The two differ by 3% in the ranking signal — inside its own resolution. Rank order is reliable in bulk and not at its boundary.

The v1 build also wasted bytes to packing padding: d=8 on down_proj gives 80 sub-vectors per row, and the packer charges whole 32-sub-vector blocks, so every row stored 42 words to carry 35. Moving down_proj to d=4/K=256 packs exactly and paid for the promotions.

KL to the bf16 teacher — the sharpest of these instruments, because it measures how far the whole output distribution moved rather than whether one ranking decision survived. Prose referee, 2048 tokens against the teacher's cached top-64 (captured mass 0.9626; same cache, same positions, every row re-measured 2026-09-15 with the corrected scorer):

v2 cuts divergence from the teacher by 13.9% against the v1 build it replaces, at the same byte budget, and gains 1.8 points of top-1 agreement.

Read this ladder by KL, and notice why. KL and top-1 order the rungs exactly as their bit budgets predict — KL roughly halves at each step up. Perplexity does not: the 3.2bpw rung reads 5.1684, better than 4.4bpw, 5.5bpw and the 178 GiB q8, while sitting at more than five times q8's divergence from the teacher. A rung chosen on perplexity alone would be chosen wrong. Perplexity is a mean over finite text and absorbs offsetting errors; KL measures the distribution itself.

Older KL figures published for this family came from a streamed scorer since found to disagree with a direct full-model forward; the two rows here were re-measured 2026-09-15 with the corrected one and are not comparable to those earlier numbers.

Run it

pip install git+https://github.com/ml-explore/mlx-lm.git@refs/pull/1788/head

python -m mlx_lm generate \
  --model TheDrainFlorist/Qwen3.8-Flash-Next-VQ-2.1bpw \
  --prompt "Explain vector quantization briefly." \
  --max-tokens 512

The VQ runtime needs no patches — it ships inside the checkpoint as model.py and stock mlx-lm executes it. The PR above is required only for the base architecture; see Requirements.

Speculative decoding (MTP)

This repo includes mtp-head-q6.safetensors (2.1 GiB): the model's own multi-token-prediction head, quantized. Stock loaders ignore it — it costs nothing on disk-to-RAM unless you opt in by name.

When enabled it adds ≈2.2 GiB resident (head weights + its cache) on top of the trunk, so budget for it. What you get, measured on this artifact (M3 Ultra, greedy, 378-token runs): 18.1 -> 24.4 tok/s, acceptance 0.77. The trunk verifies every drafted token by exact rejection sampling, so the output distribution is exactly the base model's. Prefill in the speculative path is chunked (2048-token chunks, VQLAB_PREFILL_CHUNK overrides) with per-chunk eval, so long prompts stay within the same memory envelope as plain decoding.

To use it, serve with VQLab:

git clone https://github.com/noahzelezny/VQLab && cd VQLab
python3 -m venv .venv && source .venv/bin/activate
pip install .
python -m vqlab.cli serve \
  --model TheDrainFlorist/Qwen3.8-Flash-Next-VQ-2.1bpw \
  --sidecar mtp-head-q6.safetensors

OpenAI-compatible API on localhost; vqlab mtp-generate for one-shot CLI use. Without --sidecar, nothing about the model changes.

Methodology

Fitted data-free from the bf16 checkpoint — k-means / Lloyd over weight subvectors, seed 1234, no Hessian, no activation statistics, no calibration corpus. Recipes are in the VQLab repo.

Quantization damage is not uniform across layers. A one-pass probe (teacher and student streamed together, per-layer local damage measured with no compounding) shows the same hot set on every rung of this family: layer 1 dominates, a late band (31–39) follows, and the map is identical across geometries (rank correlation 0.905, identical top-10). Upgrading only those layers buys 15–24% KL for 3–4% size on the lower rungs; the probe, the mixing, and the verdicts are all reproducible with VQLab (vqlab layer-leverage, scatter fits via fit-moe --vq-layers).

Verification

Release gates passed on this artifact before upload: file, index and tokenizer checks, a verbatim match between the bundled runtime and its source, and a generation smoke through the shipping runtime on Apple Silicon. The upload path runs the gate itself and refuses to publish without it.

Limitations

• —This model needs an unreleased `mlx-lm`. The qwen4_exp architecture is in PR #1788, still unmerged as of 2026-09-09. See Requirements.
• —Stock exo cannot serve it — upstream pins a released mlx-lm that lacks the architecture. Use our fork's mtp-stage1 branch.
• —Rank the table by KL, not perplexity. Perplexity is an aggregate over finite text and absorbs offsetting errors; several rungs read within noise of the teacher on perplexity while differing by an order of magnitude in KL.
• —The affine comparators are our own conversions, not community builds.
• —Perplexity is not comparable across model families — only within this table, which is one instrument on one corpus set.

Paper

The method, the full model ladder, the negative results, and the measurement rules behind every number here: **Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits** (CC BY 4.0) · code: VQLab · web version: Space

Provenance

Base model: Qwen/Qwen3.8-Flash-Next — released under the Qwen Community License 1.0. This is a quantized derivative and inherits that licence; using it means accepting the base model's terms. Quantization: TheDrainFlorist, 2026.

Local artifact: qwen4exp_vq_packed_mixL01.

Built with MLX and VQLab.