letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound

MiMo-V2.6-Distill-Qwen-9B — INT8 W8A16 (AutoRound)

8-bit weight-only quantization of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B produced with AutoRound 0.14.2 in the auto_round:auto_gptq layout, so it loads through the GPTQ-Marlin int8 kernels in SGLang / vLLM. Same architecture, tokenizer and chat template as the source; 12.89 GB across 8 shards.

This is the high-fidelity sibling of letechlead/MiMo-V2.6-Distill-Qwen-9B-INT4-W4A16-AutoRound: INT4 for throughput, INT8 when you want the quantization to be effectively invisible (or when the model has to fit a smaller card).

Quantization

Kept in BF16 on purpose:

• —Vision tower (model.visual.*, 333 tensors) — AutoRound's inference backend does not quantize visual blocks, so the whole tower stays BF16 and is declared bits: 16 in quantization_config.extra_config.
• —GDN gating projections linear_attn.in_proj_a / in_proj_b (48 projections) — 32-wide outputs cannot be Marlin-repacked (size_n % 64 != 0), so they are stored as plain BF16 weights and marked bits: 16.
• —Embeddings / lm_head / norms / biases — standard practice, and required by the same extra_config entries.

Quantized: the MLP (gate_proj / up_proj / down_proj), the linear-attention projections (in_proj_qkv, in_proj_z, out_proj) and the full-attention projections (q_proj, k_proj, v_proj, o_proj).

Serving

SGLang v0.5.20 (built from source, commit 94602c9c2b7cbdb8efd5c52802dac6a1c180089e), 2× RTX 3090, tensor parallel 2:

python3 -m sglang.launch_server \
  --model-path letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound \
  --served-model-name MiMo-V2.6-Distill-Qwen-9B-INT8-AutoRound \
  --tp-size 2 \
  --reasoning-parser mimo \
  --tool-call-parser mimo \
  --context-length 262144 \
  --mem-fraction-static 0.85 \
  --kv-cache-dtype fp8_e5m2

--reasoning-parser mimo is required for the model's reasoning output; the source card also recommends --tool-call-parser mimo for tool use. Verified on SM86 (RTX 3090): the artifact loads through SGLang's auto_round → GPTQ-Marlin int8 path without patching, and answers correctly.

Long context on 2× RTX 3090

The model's native window is 262,144 tokens with no rope scaling, and the whole window runs on the pair:

• —Only 8 of 32 layers carry a KV cache; the other 24 are GatedDeltaNet linear attention with constant-size state. KV is therefore 32 KiB/token at fp16 and 16 KiB/token with the 8-bit (fp8_e5m2) cache.
• —Measured pool: 933,890 tokens (3.6× the model max), max_req_input_len 262,138.
• —Weights + cache fit at ~22.2 GiB per card with --mem-fraction-static 0.85.
• —Prefill is chunked at 16,384 tokens so a long prompt does not stall the scheduler.
• —The GDN state cache caps concurrent requests at roughly 59; the window is limited by linear-attention state, not by the KV pool.

Quality (measured, not assumed)

Checkpoint-level coherence against the BF16 source — 8,192 tokens of wikitext-2-raw-v1 test, 15 windows of 1,024 with stride 512, 15,345 scored positions, float32 logits, identical token ids for both models:

Per-window agreement 97.3–98.8%, per-window KL 0.0012–0.0136. The negative perplexity delta is within noise: at 8-bit the quantization is effectively invisible on this corpus, and it is 20× closer to the BF16 distribution than INT4 (KL 0.004 vs 0.084).

Behavioural probe (7 prompts: arithmetic, factual recall, letter counting, sequence completion, two instruction-following constraints, one open prose question), graded against the same prompts on the BF16 source:

INT4 misses the bat-and-ball trap and an "exactly two sentences" constraint; INT8 gets both.

Throughput

llama-benchy 0.3.5, pp 4096 / tg 512, 3 runs each, cold, concurrency 1, exact tokenizer, 2× RTX 3090 TP=2, 262,144 ctx with the 8-bit KV cache:

All 7 depths × 3 runs completed, no failures. The 257,000 rung is a 261,096-token prompt.

Against INT4 at the same configuration:

Decode costs 17–25% because decode is weight-bandwidth-bound and int8 doubles the weight bytes; prefill and TTFT are essentially unchanged (1746 vs 1766 tok/s at 257K) because prefill is compute-bound.

Which one to use

• —INT8 (this artifact) — when quantization must not change model behaviour, or when 12.89 GB fits where 18.85 GB BF16 does not (a single 24 GB card leaves ~11 GB for cache instead of ~5 GB; 16 GB cards and CPU-offload setups cannot hold BF16 at all).
• —INT4 W4A16 — when you want maximum decode speed and a smaller artifact, and can accept the measured quality delta (+1.36% perplexity, 87.5% top-1 agreement, 5/7 on the probe).
• —BF16 — when the pair has room and you want the exact source distribution.

Verification performed

• —760/760 source tensors accounted for; 0 missing, 0 extra, 1,160 tensors total.
• —Every quantized linear has its full qweight / qzeros / scales triple.
• —No quantized output width fails % 64; no quantized output is narrower than 64.
• —Group packing aligned (in-features divisible by 128/pack-factor).
• —Loads and serves in SGLang v0.5.20 on SM86 with the GPTQ-Marlin int8 path; smoke test correct.
• —Tokenizer identity: tokenizer.json and chat_template.jinja are byte-identical to the BF16 source, and the served tokenizer returns the same ids as both the artifact's own tokenizer and the source for identical inputs via /tokenize (English, Arabic, code). tokenizer_config.json differs from the source in one field only: tokenizer_class is TokenizersBackend as written by AutoRound, where the source has Qwen2Tokenizer.
• —Coherence and behavioural numbers above are measured on this artifact, not carried over.

Files

model-0000{1..8}-of-00008.safetensors   int8 weights + BF16 kept tensors
model.safetensors.index.json            1160 tensors, 8 shards
config.json                             model config + embedded quantization_config
quantization_config.json                bits 8, g128, sym, auto_round:auto_gptq, extra_config
tokenizer.json / tokenizer_config.json / chat_template.jinja
preprocessor_config.json / processor_config.json / generation_config.json

Attribution

Base model: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (MIT). Quantization: AutoRound 0.14.2. Serving stack: SGLang v0.5.20. Quantized and verified by LeTechLead.