ngquocvinh/MiMo-V2.6-Distill-Qwen-9B-GGUF

MiMo-V2.6-Distill-Qwen-9B GGUF

Community GGUF quantizations of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B.

<div align="center" style="background-color:#f59e0b;color:#ffffff;padding:16px 20px;border-radius:10px;line-height:1.7;"> ☕ If this GGUF made your day easier, a coffee would make mine.<br> <a href="https://ko-fi.com/ngquocvinh" style="color:#ffffff;"><strong style="color:#ffffff;">Send a coffee ☕</strong></a><br> I build and test these releases myself. Your coffee helps keep me going.<br> Thank you for supporting this work. </div>

About MiMo-V2.6-Distill-Qwen-9B

MiMo-V2.6-Distill-Qwen-9B is a 9B agentic supervised-fine-tuning checkpoint from Xiaomi MiMo, based on Qwen3.5-9B. The upstream card describes coding, general-purpose agent tasks, visual coding, and cybersecurity. This GGUF package contains quantized language-model files and a matching BF16 multimodal projector for image input.

The converted language model preserves the upstream hybrid Qwen3.5 architecture and 262,144-token native context metadata. Practical context length depends on the selected quantization, KV-cache settings, runtime, and available memory. The image path was validated with llama.cpp; video behavior is not claimed by this release.

Fidelity measurements

These are next-token fidelity measurements against the converted BF16 language-model reference. They use four chunks from wiki.valid.raw at context length 2048, with the same qwen4exp llama.cpp llama-perplexity runtime for every row. Values are averaged across the evaluated chunks. Lower Mean KLD, ΔPPL, and RMS Δp, and higher Top-1 agreement, indicate closer next-token behavior to BF16. These measurements are not task, coding, tool-use, or visual-understanding benchmarks.

The bold rows are the current sweet spots: they offer a practical memory/fidelity balance while retaining high Top-1 agreement with BF16. Sizes use decimal GB (bytes / 1e9) and are also recorded in reproducibility/quantization-manifest.tsv. Exact VRAM fit depends on context length, KV-cache settings, offload, and runtime, so this is not a fixed VRAM guarantee. The remaining low-bit files are lower-memory or experimental profiles and are not preferred when fidelity matters.

The negative ΔPPL values for Q5KM, Q4KM, and Q2_K do not mean that quantization improves the model; they are measurements of this fixed small hold-out and should be interpreted together with KLD, Top-1 agreement, and RMS Δp.

The compact machine-readable table is `reproducibility/quality-summary.tsv`. Commands, hashes, source split information, and validation details are in `reproducibility/manifest.md`.

Quick start

Text generation with Q4KM:

./llama-cli \
  -m MiMo-V2.6-Distill-Qwen-9B-Q4_K_M.gguf \
  --chat-template-file chat_template.jinja \
  --jinja \
  --reasoning off \
  -p 'Answer briefly in English: What is GGUF and why is it useful for running language models locally?' \
  -n 128 -c 4096 -ngl 99

Image input with the BF16 projector:

./llama-mtmd-cli \
  -m MiMo-V2.6-Distill-Qwen-9B-Q4_K_M.gguf \
  --mmproj mmproj-MiMo-V2.6-Distill-Qwen-9B-BF16.gguf \
  --image image.jpg \
  --jinja \
  -p 'Describe the visible objects and their spatial relationship in this image in one concise English paragraph.' \
  -n 128 -c 4096 -ngl 99

Adjust GPU offload, context, batch size, and KV-cache types for the target machine. The Q2/IQ2/IQ1/Q1 files are experimental and should not be used when capability retention is important.

Reproducibility and validation

• —Upstream repository: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
• —Locked upstream revision: f2773fb482ac3dd047a4af4003b86e56b7225d0d
• —Source format: upstream BF16 safetensors; no quantized model was used as input
• —Converted BF16 language-model GGUF and BF16 projector were produced directly from that source
• —All nine language-model files were quantized directly from the same BF16 language-model GGUF
• —The imatrix is model-specific and combines WikiText-2 train text with EN/ZH/code/tool/agent calibration text
• —wiki.valid.raw and wiki.test.raw were not used to create the imatrix
• —All nine language files passed deterministic text load/generate smoke tests
• —Q4KM passed an image smoke test with the matching BF16 projector
• —Artifact checksums are in `SHA256SUMS.txt`

Raw conversion, imatrix, quantization, smoke-test, and fidelity logs remain local under reports/ and are not part of this package.

License and attribution

The upstream MiMo model repository did not provide a license declaration at the time of this build. Public redistribution is therefore intentionally blocked until Xiaomi MiMo's applicable license terms are confirmed. The temporary license: other metadata must not be treated as a grant of rights.

These are community GGUF quantizations, not an official Xiaomi MiMo release or endorsement.