Modelpublic

Kuberwastaken/Kimi-K3-GGUF

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

14likes627downloads

Model Card

What this is

GGUF conversion assets for [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3) — Moonshot AI's 2.8T-parameter (104B active) multimodal MoE.

This repo hosts the complete vision encoder, the full tokenizer and architecture metadata, and a converter proven lossless across the entire 1.419 TiB model. It does not re-host the expert weights: those already live in Moonshot's repo, and the converter streams from there so you get identical bytes without a second terabyte sitting on the Hub.

[!WARNING] Nothing here runs yet. llama.cpp has no kimi_k3 architecture, so no GGUF of this model — from anyone — will load in llama-cli today. These files are correct, verified and ready for the moment support lands. See What's blocking it; the gap is smaller than the model's size suggests.

File	Size	What it is
`mmproj-Kimi-K3-BF16.gguf`	853 MiB	Complete 27-layer MoonViT vision tower + `patchmergerv2` projector, lossless BF16
`Kimi-K3-tokenizer.gguf`	6.6 MiB	163,840-token vocab, 163,328 merges, all `kimi-k3.*` hyperparameters — no tensors
`MANIFEST.json`	458 KiB	SHA-256 of all 2,760 output tensors, so you can verify your own conversion
`convert_kimi_k3.py`	—	Streaming, resumable text-model converter
`convert_vision.py`	—	Vision tower → mmproj
`convert_tokenizer.py`	—	tiktoken → GGUF vocab
`modal_run.py`	—	Cloud fan-out (94 containers, ~5 min wall-clock)
`k3lib.py` · `k3meta.py` · `k3vocab.py`	—	MXFP4 repack, arch metadata, vocab reconstruction
`TENSOR_MAP.md`	—	Every tensor: HF name → GGUF name, with transforms

Setup

bash

pip install numpy gguf huggingface_hub
hf download Kuberwastaken/Kimi-K3-GGUF --local-dir k3gguf
cd k3gguf

That's enough to use the vision encoder, tokenizer and manifest. To build the full model you additionally need the metadata files from the source repo:

bash

hf download moonshotai/Kimi-K3 --local-dir meta \
  config.json model.safetensors.index.json tokenizer_config.json tiktoken.model

Requirements for a full conversion


Disk (working)	~32 GiB — one input shard + one output shard
Disk (final)	~1.42 TiB for the finished GGUF
RAM	~6 MiB of tensor data at a time (experts stream through a memmap)
Time	~4 min per shard; 94 shards, fully parallelisable
GPU	none — this is a byte-level repack, not a compute job

There is no fp16 intermediate at any point. A dequantise-then-requantise pipeline would need ~5.6 TB of scratch; this needs one shard.

Operating guide

Convert the whole model

bash

python convert_kimi_k3.py --meta ./meta --out-dir ./out --shards 1-94

Output is Kimi-K3-MXFP4-000NN-of-00094.gguf. Weights stream from the Hub automatically; pass --src ./local-copy if you already have the safetensors.

Convert a piece at a time

Source shard k holds *exactly layer k−1***, and no tensor spans two shards. The 94 units are therefore fully independent — interrupt, resume, or spread them across machines freely.

bash

python convert_kimi_k3.py --meta ./meta --out-dir ./out --shards 2      # layer 1
python convert_kimi_k3.py --meta ./meta --out-dir ./out --shards 2,5,9  # a few
python convert_kimi_k3.py --meta ./meta --out-dir ./out --shards 40-60  # a range

Shard	Holds
`1`	layer 0 (dense MLP) + all model KV metadata + tokenizer
`2`–`93`	layers 1–92, one per shard, 896 experts each
`94`	`token_embd`, `output`, `output_norm`, AttnRes output tensors

Options

Flag	Default	Meaning
`--meta`	`src`	Directory with `config.json`, index, `tiktoken.model`
`--src`	(stream)	Local safetensors copy; omit to pull from the Hub
`--out-dir`	`out`	Where GGUF shards are written
`--shards`	`1-94`	`N`, `A-B`, or a comma-separated mix
`--scratch`	(out-dir)	Expert memmap scratch; point at a fast disk

In the cloud

modal_run.py fans the same work across 94 containers (~$4 of Modal credits, ~5 min wall-clock). Create a huggingface secret holding HF_TOKEN first — 94 unauthenticated containers will hit Hub rate limits.

bash

modal secret create huggingface HF_TOKEN=hf_...
modal run modal_run.py::pilot            # one layer, verified, no upload
modal run modal_run.py::convert_all      # all 94
modal run modal_run.py::build_manifest   # all 94 + SHA-256 manifest

Rebuild the small artifacts

bash

python convert_vision.py    --src ./meta --out mmproj-Kimi-K3-BF16.gguf
python convert_tokenizer.py --src ./meta --out Kimi-K3-tokenizer.gguf

convert_vision.py needs shards 95 and 96 (the projector and vision tower).

Why the conversion is lossless

Kimi-K3 ships natively in MXFP4 (compressed-tensors, mxfp4-pack-quantized, group size 32, E8M0 uint8 scales). GGML's block_mxfp4 stores exactly the same information:

source:  packed nibbles [K/2 bytes]  +  E8M0 scale [K/32 bytes]
ggml:    block_mxfp4 { uint8 e; uint8 qs[16]; }  ×  K/32

For one expert tensor that is 5,505,024 + 344,064 = 5,849,088 bytes on both sides — identical. The conversion is a nibble permutation (the source packs consecutive pairs 2k/2k+1; GGML packs j/j+16) with the scale byte copied verbatim. No dequantisation, no intermediate, no precision loss.

The scale byte transfers untouched because GGML's half-scale × doubled-kvalues convention is algebraically identical to the OCP MX interpretation: E8M0_TO_FP32_HALF(e) × (2·e2m1) == 2^(e-127) × e2m1.

[!NOTE] One tensor's values do change. A_log → ssm_a is stored pre-transformed as -exp(A_log), matching llama.cpp's kimi_linear.py. Every other tensor in the model is bit-preserved.

Verification

Nothing above is asserted without a check:

What	Result
MXFP4 repack vs. `gguf-py`'s own dequantizer, real K3 tensor	0 mismatches / 11,010,048 elements, max abs diff `0.0`
Source vs. GGML byte counts, expert tensor	`5,849,088 == 5,849,088`
`mmproj` vs. source safetensors	168/168 tensors byte-identical, shapes correct
Layer 0 shard vs. source	22/22 byte-identical (2,232 MiB), `ssm_a == -exp(A_log)` exact
Layer 1 shard, 896 stacked experts	0 mismatches, 15.82 GiB, 31 tensors
Full model — 94 shards, 1.419 TiB streamed	0 mismatches, 2,760/2,760 tensors

Two independent implementations are compared for the MXFP4 path: a NumPy reference written from the compressed-tensors semantics, and llama.cpp's own gguf.quants.MXFP4 dequantizer. They agree bit-for-bit.

Verify your own conversion

MANIFEST.json records the SHA-256 of the canonical GGML bytes for all 2,760 tensors, with ggml type and ne:

json

"blk.1.ffn_down_exps.weight": { "sha256": "…", "type": 39, "ne": [3072, 3584, 896] }

So you never have to take this on faith:

python

import hashlib, json
from gguf import GGUFReader

man = json.load(open("MANIFEST.json"))["tensors"]
for t in GGUFReader("out/Kimi-K3-MXFP4-00002-of-00094.gguf").tensors:
    assert hashlib.sha256(t.data.tobytes()).hexdigest() == man[t.name]["sha256"], t.name
print("match")

Composition: 1,978 BF16 + 506 F32 + 276 MXFP4 (stacked experts, 92 layers × 3) = 2,760 tensors, 1,559,972,708,032 bytes. The ~846 MiB difference from the source index is the vision tower, which lives in the mmproj file instead.

Warnings and gotchas

[!WARNING] It will not load in llama.cpp yet. general.architecture is kimi-k3, which yields an honest "unknown architecture" error. Declaring kimi-linear would make llama.cpp attempt a graph that cannot represent this model.

—You still need the disk. GGUF has no remote-pointer mechanism — tensor bytes live inside the file. Streaming avoids re-hosting, not downloading.
—Don't run 94 containers unauthenticated. Set HF_TOKEN or you'll hit Hub rate limits partway through.
—`--scratch` on a slow disk will dominate runtime. Each layer stages a ~5.2 GiB expert memmap.
—Tensor names are a considered guess for K3-only tensors. AttnRes and LatentMoE have no upstream precedent yet; whoever lands the kimi_k3 PR picks the final names. Everything with a kimi-linear equivalent already uses it. Full table in `TENSOR_MAP.md`.
—`gguf-py` shape trap. Passing a uint8 buffer with raw_dtype makes gguf-py treat raw_shape as a byte shape and divide by the type size, silently halving the declared row length. Pass a 2-byte view for BF16.

What's blocking it

llama.cpp has no kimi_k3 architecture — but it already ships LLM_ARCH_KIMI_LINEAR with a full llama_model_kimi_linear implementation, and K3's text_config.architectures is literally ["KimiLinearForCausalLM"]. Kimi Delta Attention and MLA are already implemented upstream.

The remaining delta is narrower than 2.8T parameters suggests:

—AttnRes — extra residual pathway (attn_res_*, ffn_res_*, output_attn_res_*), attn_res_block_size = 12
—LatentMoE — routed experts run in a 3,584-dim latent space, not model space
—`situ` activation — β = 4.0, linear β = 25.0
—2 shared experts (Kimi-Linear has 1) and 896 routed, 16 active
—output gate on every layer (mla_use_output_gate)
—NoPE — mla_use_nope = true, no rope_theta anywhere in the config
—the multimodal wrapper (KimiK3ForConditionalGeneration)

Tensor names here deliberately follow the existing kimi-linear conventions so that delta stays as small as possible.

Model facts


Parameters	2.8 T total / 104 B active
Layers	93 — 24 MLA + 69 KDA
Hidden size	7,168
Experts	896 routed (16 active) + 2 shared
Expert latent dim	3,584
Vocabulary	163,840 (tiktoken, `kimi-k2` pre-tokenizer)
Context	1,048,576
Vision	27-layer MoonViT, patch 14, `sd2_tpool` merge
Native precision	MXFP4 experts, BF16 everything else

Credits

Model and logo by [Moonshot AI](https://huggingface.co/moonshotai) — the weights this repo converts live at [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3). Conversion conventions follow llama.cpp's kimi_linear.py and gguf-py. Licensed under the upstream Kimi-K3 License.

Built with ♥ by Kuber Mehta (<a href="https://kuber.studio">kuberwastaken</a>)

What this is

Contents

Setup

Requirements for a full conversion

Operating guide

Convert the whole model

Convert a piece at a time

Options

In the cloud

Rebuild the small artifacts

Why the conversion is lossless

Verification

Verify your own conversion

Warnings and gotchas

What's blocking it

Model facts

Credits