Modelpublic

islamassanov/Kimi-K3-Mixed-IQ2-GGUF

sourceHugging Faceotherupdated 7d agoView on Hugging Face

1likes925downloads

Model Card

Kimi K3 2-bit Mixed-IQ2 GGUF

A downloadable, multimodal GGUF quantization of MoonshotAI/Kimi-K3. The routed experts use a mix of IQ2XXS and IQ2XS; the vision projector remains BF16.

At a glance
Base model	2.8T-parameter MoE; 16 of 896 routed experts active per token
Text weights	793 GiB in 10 GGUF shards
Vision projector	862 MiB, BF16
Context	Up to 1M tokens in the base architecture; usable context depends on available memory
Integrity	Per-file SHA-256 hashes in `verification-receipt.json`

Quick start

Download the full repository and load the first shard with a recent llama.cpp build:

bash

hf download islamassanov/Kimi-K3-Mixed-IQ2-GGUF \
  --local-dir Kimi-K3-Mixed-IQ2-GGUF

llama-server \
  -m Kimi-K3-Mixed-IQ2-GGUF/Kimi-K3-Mixed-IQ2-00001-of-00010.gguf \
  --mmproj Kimi-K3-Mixed-IQ2-GGUF/mmproj-Kimi-K3-BF16.gguf \
  -c 32768

llama.cpp finds the other nine shards automatically. Omit --mmproj for text-only use. Allow roughly 794 GiB for the files, plus memory for model weights, KV cache, and runtime buffers.

What I changed

Kimi K3's released expert weights are already MXFP4, so this is requantization of the released checkpoint, not quantization from BF16. I encoded 146,048 routed expert matrices as IQ2XXS and upgraded 101,248 to IQ2XS. The upgrades were chosen by measured reconstruction-error reduction within a fixed byte budget. Another 2,628 tensors stay at higher precision.

The key encoder change is a decoder-aligned IQ2_XXS scale fit. Stock IQ2_XXS fits scales using idealized magnitudes 1, 3, 5; its decoder reconstructs 1, 3.125, 5.375. I fit the scales to those actual decoder values. The GGUF format, decoder, block size, and inference cost are unchanged.

Against the stock IQ2_XXS encoder, this reduced source-weight reconstruction error by 5.57% at identical bytes on a frozen sample of 1,104 expert matrices spanning all 92 routed layers. Every sampled matrix improved. This is a weight-fidelity result, not a percentage gain in model accuracy.

Model evaluation

This GGUF scored 871 / 1,000 (87.1%) on OCRBench — 97.9% of [Moonshot's published 89.0% base-model score](https://github.com/MoonshotAI/Kimi-Vendor-Verifier#k3-evaluation-results).

Files and license

All 10 numbered text shards are required. File sizes and hashes, including the separate vision projector, are recorded in `verification-receipt.json`.

Use of this model is subject to the Kimi K3 license. Thanks to Moonshot AI for Kimi K3 and the `llama.cpp`/GGML maintainers for the GGUF and IQ2 tooling. Questions or corrections: islam@uni.minerva.edu.