letechlead/MiMo-V2.6-Distill-Qwen-9B-INT4-W4A16-AutoRound

MiMo-V2.6-Distill-Qwen-9B — INT4 AutoRound

INT4/W4A16 AutoRound quantization of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B, a Qwen3.5 hybrid (GatedDeltaNet linear attention + full attention) image-text-to-text model distilled from Qwen/Qwen3.5-9B.

The language tower is INT4. The vision tower stays in its original BF16 precision.

Quantization

• —AutoRound: 0.14.2
• —Weight format: INT4 / W4A16
• —Group size: 128
• —Symmetric quantization
• —Packing format: auto_round:auto_gptq
• —Calibration dataset: NeelNanda/pile-10k
• —Calibration samples: 128
• —Calibration sequence length: 512
• —Quantized block: model.language_model.layers (the language tower)
• —Model type: Qwen3.5 multimodal conditional generation

Two deliberate exceptions

1. 1.Vision tower stays BF16. The quantized block covers the language tower only, so the image encoder and merger are unchanged from the source checkpoint.
2. 2.The GDN gating projections stay BF16. in_proj_a and in_proj_b of each linear-attention layer are 32-wide. Marlin's GPTQ repack kernel requires size_n % 64 == 0, so those layers cannot be served as INT4 at all; they are stored as plain BF16 weights and marked bits: 16 in the quantization config. This also matches the runtime's own GDN path, which wants the small gating projection in bf16. They are 0.1% of the model.

Layout

The checkpoint is stored in the canonical layout: every tensor carries the source checkpoint's own name (model.language_model.*), the shard index matches the files on disk, and no duplicate/extra tensor file is shipped. 8.98 GB total.

Serving

The model card requires SGLang with Qwen3.5 support. Verified with SGLang built from upstream sgl-project/sglang tag v0.5.20 (commit 94602c9c2b7cbdb8efd5c52802dac6a1c180089e) on 2× RTX 3090 (SM86):

python3 -m sglang.launch_server \
  --model-path letechlead/MiMo-V2.6-Distill-Qwen-9B-INT4-W4A16-AutoRound \
  --served-model-name MiMo-V2.6-Distill-Qwen-9B-INT4-AutoRound \
  --tp-size 2 \
  --reasoning-parser mimo \
  --tool-call-parser mimo \
  --context-length 262144 \
  --mem-fraction-static 0.85 \
  --kv-cache-dtype fp8_e5m2

The mimo reasoning parser fills reasoning_content and the mimo tool parser emits tool_calls; both were exercised through /v1/chat/completions.

Long context on 2× RTX 3090

The model supports 262,144 tokens natively (max_position_embeddings, no rope scaling), and that full window is served on a single 3090 pair. It is affordable because only 8 of the 32 layers carry a KV cache — the other 24 are GatedDeltaNet linear attention with a constant-size state:

• —KV per token: 32 KiB at fp16, 16 KiB with the 8-bit (fp8_e5m2) KV cache above
• —KV pool: 1,048,762 tokens = 4.0× the model maximum, on ~22.5 GiB per card
• —max_req_input_len 262,138; prompts longer than 16,384 tokens use chunked prefill
• —The GatedDeltaNet/mamba state cache (7.03 GB per rank) caps concurrent requests at 59 — long-context concurrency here is limited by linear-attention state, not by the KV pool

Transformers loading of this artifact is supported through auto-round's integration (AutoModelForImageTextToText.from_pretrained, with auto-round installed).

Verification

Checkpoint-level: BF16 source vs this INT4 artifact

• —Method: 8,192 tokens of wikitext-2-raw-v1 (test), windows of 1,024 with stride 512 (15 windows, 15,345 scored positions), float32 logits, identical token ids and tokenizer (vocab 248,044) for both models.
• —Perplexity: BF16 12.172 vs INT4 12.337 → +1.36%
• —Top-1 next-token agreement: 87.46% (13,421 / 15,345 positions)
• —Mean KL(BF16 ‖ INT4): 0.0842

Behavioral probe: 7 fixed prompts, greedy, thinking disabled

INT4 5/7 vs BF16 7/7. Arithmetic (391), factual recall (Canberra, Ottawa), sequence completion (32, 64, 128), token counting (3 rs in "strawberry"), the single-sentence constraint and the open-ended prose answer were all correct in both.

The two INT4 misses, reported as observed:

• —Bat-and-ball trap — INT4 answered 0.10; BF16 also opened with the wrong $0.10 but self-corrected to $0.05 inside its reasoning.
• —"exactly two sentences" — INT4 answered in one sentence (content correct).

Re-run with the 8-bit KV cache in place: 5/7 again, the same two misses — no measurable behavioral regression from KV quantization on this probe.

Throughput: llama-benchy 0.3.5, pp 4096 / tg 512, 3 runs each, cold, concurrency 1

Tokenizer = this artifact's own (verified token-identical to the served model, 3/3 test strings including Arabic). Means over 3 runs; TTFT is the measured time-to-first-byte in api latency mode. 262,144 context with the 8-bit KV cache.

All 7 depths × 3 runs completed without failures; the 257,000 rung is a 261,096-token prompt. Prefill stays above 1,766 tok/s across the ladder, while decode falls 24% from 123.0 tok/s at 8K to 93.1 tok/s at 257K.

2× RTX 3090, TP=2, 262,144 context, --mem-fraction-static 0.85, ~22.5 GiB per card.

Attribution and license

This is a derived quantized artifact of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B, released under the upstream MIT license. Please review the upstream model card for intended use, limitations, and full attribution.