GrEarl/Kimi-K3-GGUF-IQ1_S

Author Note

The author does not own enough hardware to run this 528.0293 GiB model. Runtime results below were contributed by independent users and have not been reproduced by the author. The reports predate the current top32/refit2 payload, so their exact revision scope is stated explicitly.

Kimi-K3 GGUF — IQ1S routed experts / Q4K eligible dense weights / F16–F32 structural tensors

Sub-2-bit GGUF conversion of the text model of `moonshotai/Kimi-K3` — a 2.8T-parameter MoE (896 experts, 16 active) shipped in MXFP4 via quantization-aware training.

⚠️ Read this before downloading 528 GiB

This does not load with any released version of llama.cpp. Kimi-K3 support is still an open pull request: ggml-org/llama.cpp#26185. You must build from that branch (pwilkin/llama.cpp:kimi-k3-text).

Hugging Face's autogenerated "Use this model" snippets (vLLM, Ollama, stock llama.cpp) are not valid for this repository. Ignore them.

Evidence levels — please do not conflate these

Independent runtime reports

Discussion #1 contains at least two independent full-generation reports:

• —One user reported local llama.cpp generation at about 19.7 tokens/s and supplied a screenshot, but did not state hardware, command line, llama.cpp commit, or repository revision. Treat the speed as an observation, not a reproducible benchmark.
• —A second user reported running on 512 GB DDR5-4800, an Intel QYFS Sapphire Rapids ES CPU, and 96 GB VRAM (1×4090 + 3×3090). They reported about 20 tokens/s prompt processing, 5 tokens/s generation, a 32K context, and completion of a long one-shot HTML generation. The launch command and exact offload split were not provided.

These reports establish that the 94-part IQ1S layout and an earlier IQ1S payload could be fully loaded and used for generation, including on a large-RAM consumer-GPU system. They remain third-party reports and were not reproduced by the author.

Revision boundary: the reports were posted/updated on 2026-07-28 UTC. The current top32/refit2 payload was promoted on 2026-07-29 at 01:52 UTC, so the reports cannot be attributed to its exact bytes. The promotion retained the same 94-part tensor contract, tensor types, offsets, metadata, and total size while replacing only IQ1_S expert payload bytes; sampled reconstruction metrics improved. That makes the reports strong compatibility evidence, but not a direct current- revision runtime or quality validation.

The second user noted that they needed to fix the chat template. Their exact revision and failure mode were not supplied. The current card therefore does not infer that the present 15,053-byte input template is broken: it is separately 28/28 byte-matched against K3 and rendered through Minja. K3 output parsing is a distinct runtime concern on branches without a dedicated parser.

What was verified, on the actual uploaded files

Read back with HTTP range requests over all 94 parts (headers and KV only):

• —All 94 parts present, split.no = 0..93, split.count = 94 everywhere
• —split.tensors.count = 2573, written as INT32, identical in every part
• —Sum of per-part tensor counts = 2573 — the value llama_model_loader compares against weights_map.size()
• —Tensor names identical to the set generated by the PR's create_tensor calls: 0 missing, 0 extra, 0 duplicates
• —Longest tensor name 29 chars (GGML_MAX_NAME is 64)
• —Part 1 carries 66 KV entries: all 25 hparams the PR's loader reads, plus the full vocabulary
• —Type distribution: IQ1S 276 / Q4K 1067 / F32 1112 / F16 117 / Q8_0 1

manifest.json in this repository lists per-file size and SHA-256 for all 94 parts.

Known limitations and unverified assumptions

• —No importance matrix. llama-quant.cpp marks IQ1_S as requiring one and refuses to produce it without ("The result will be garbage, so bailing out"). The runtime reports show that collecting activation statistics is now possible in principle, but no imatrix/calibration run was used for this artifact. This remains a real quality limitation.
• —The strongest tested codebook search is applied: stable top-32 exact-objective candidates with two scale-refit rounds (see Codebook search below).
• —Metal / Vulkan. The cross-layer residual uses ggml_dsv4_hc_pre, which has CPU and CUDA implementations only. Other backends are expected to take the scheduler's per-node fallback path; the graph contains 187 such nodes, so decode would incur many device round trips. The reports above establish generation for an earlier payload on llama.cpp setups that included GPU acceleration; they do not verify the current top32/refit2 bytes, Metal, Vulkan, or other backends.
• —Chat template: normal chat, thinking_effort, tool declarations/calls/results, and response_format / response_schema are covered. Images and batched conversations remain untested. See the tokenizer section.

Quantization mix

Routed experts hold 97.9% of the parameters, so they set the file size.

The policy follows upstream's own rules in llama-quant.cpp: names not ending in weight are never quantized, nor are *_norm.weight, ffn_gate_inp.weight, or ssm_conv1d*. Row lengths were checked against block size for all 2,573 tensors.

Weight-space reconstruction error — not end-to-end model quality

These measurements do not establish generation quality, perplexity, logit fidelity, or routing fidelity. They compare quantized tensors against the MXFP4 source in weight space only. The source is already 4-bit with 21 distinct values per group, so this is a 4-bit → 1.5-bit requantization.

Full methodology and distributions are in quantization-report.json.

Format comparison on one tensor

layers.48.block_sparse_moe.experts.0.w1, all 11,010,048 elements, rel_rmse = sqrt(mean((Q-W)²))/std(W), cos = dot(W,Q)/(‖W‖‖Q‖):

† IQ1M and TQ10 were measured with the older codebook selection and were not re-measured after the fix. Their numbers are therefore not directly comparable to the current IQ1_S row; both would improve by an unknown amount.

Format choice. Under the old selection, TQ10 was Pareto-dominated by IQ1S (larger and worse) while IQ1M was **not** dominated — it traded about 12% more expert storage (~587 GiB total, +59 GiB) for slightly lower error. With stable top-32 selection and two refits, IQ1S at 1.5625 bpw reaches 0.890112 on the representative tensor, better than the superseded top-8 result without changing a single file byte in size. A fixed IQ1_M would presumably move ahead again on error; it was not built, because the objective was the smallest operating point around 530 GiB. This is stated as a size/error trade-off, not as domination.

Distribution across layers

Sampled during conversion: one tensor per MoE layer (block_sparse_moe.experts.0.w1), all elements measured, 92 of 92 MoE layers.

The spread genuinely is this narrow — the tails are reported so this can be checked rather than assumed.

Sampling gaps, stated explicitly: only w1 was measured, only expert index 0, i.e. 1 of 2,688 expert tensors per layer (0.037%). w2 (input axis 3072) and w3 were not measured in production. Tail behaviour across experts within a layer is unmeasured. With 1.5-bit weights, a handful of outlier tensors could matter, and this sampling would not see them.

Codebook search: what was fixed, and what is still left

The codebook search minimises Σ(dl·(g+δ) − x)². Substituting u = x/dl − δ gives dl²·Σ(g−u)², so candidates must be ranked against the unrounded u. The original artifact used a top-8 shortlist and one scale refit. The current artifact uses a stable top-32 shortlist ranked by the exact objective, preserves the exact in-grid skip, and performs two scale-refit rounds.

Measured on the representative tensor:

The current files reduce mean reconstruction error by 15.2% at no change in file size (0.5375 → 0.455936 across 92 layers). Two properties make this efficient:

• —clip(round(u)) is the exact nearest point of the lattice {-1,0,1}⁸, and the grid is a subset of that lattice. So when the rounded code is one of the 2048 grid codes, the old snap was already optimal and no search is needed. That is about 48% of groups, skipped losslessly.
• —Refitting the scale after choosing the codebook entry is cheap relative to the candidate evaluation. The current files use two refit rounds after stable top-32 selection and reach 0.890112 cosine on the representative tensor.

The current artifact uses the strongest tested 1.5625-bpw operating point. It still does not reach Q2_K's 0.951 cosine; this remains an aggressive 1.5-bit format.

An importance-matrix proxy that was tried and rejected

Since an imatrix requires running the model, an analytic substitute was derived: for a tensor fed directly by an RMSNorm, E[x_i²] ∝ w_norm_i². The routed experts are not fed by a norm — the reference implementation applies routed_expert_norm to the expert output — so the importance was propagated as v[j] = Σ_i W_down[j,i]² · ffn_norm_i². The result is nearly flat (p99/p1 = 1.1×), because summing 7168 positive terms concentrates, and weighting by it produced no improvement.

This shows the proxy is unusable, not that importance weighting would not help. The diagonal approximation discards exactly the anisotropy being sought (the off-diagonal of the input covariance).

Quantizer verification

llama.cpp cannot yet read this architecture, so llama-quantize could not be used. The published expert payloads were generated by the fused stable-top-32 implementation and cross-checked against the authoritative NumPy path and gguf-py 0.17.1 dequantization:

Two details worth recording:

• —GGML rounds with lroundf (away from zero); NumPy's np.rint is banker's rounding. Every MXFP4 value is a dyadic rational, so x/scale lands exactly on .5 often and the difference changes output bytes. TQ1_0 byte equality only appeared after matching this.
• —A naive IQ1_S search scans 2048 codebook entries per 8 elements. Since a ternary 8-tuple has only 3⁸ = 6561 forms, a one-time "6561 → nearest grid index" table makes it O(1) per group. The same table also supplies the ranked candidate list used by the fixed search, and marks which codes need no search at all.

Conversion details

The file records what was done to it:

kimi-k3.conversion.contract         = llama.cpp PR #26185 (pwilkin/kimi-k3-text)
kimi-k3.conversion.source_revision  = 9f62e4e9fffbd0a83ddd60e1c209d828994b3569
kimi-k3.conversion.source_quant     = compressed-tensors/mxfp4-pack-quantized
kimi-k3.conversion.expert_quant     = IQ1_S
kimi-k3.conversion.dense_quant      = Q4_K
kimi-k3.conversion.a_log_transform  = -exp(A_log[:n_head])
kimi-k3.conversion.attn_res_fused   = res_norm*res_proj[0] in float32
kimi-k3.conversion.kv_b_split       = k_b(transposed)+v_b
kimi-k3.conversion.expert_stack_dim = 0
kimi-k3.conversion.vision_excluded  = true
kimi-k3.conversion.imatrix          = false
kimi-k3.conversion.defaults_used    = rope_theta

Notable transforms:

• —`ssm_a` = `-exp(A_log[:n_head])`. K3 stores A_log with shape [head_dim] = 128, but only the first num_heads = 96 entries are used and the loader expects 96. Without the narrowing: check_tensor_dims: tensor 'blk.0.ssm_a' has wrong shape; expected 96, got 128. The model's own bundled modeling_kimi_linear.py declares A_log with num_heads elements while the shipped weight has head_dim; the weight is authoritative and only its first 96 entries matter.
• —AttnRes is fused. <x>_res_norm.weight * <x>_res_proj.weight[0] folded into one float32 vector per site (attn_res_score, ffn_res_score, output_res_score). Exact, not an approximation: the reference _apply_attn_res() computes the same product itself. Removes 2 × 93 + 1 = 187 tensors (2,760 → 2,573).
• —`kv_b_proj` split into attn_k_b (transposed) and attn_v_b, the path the loader requires when the unsplit tensor is absent.
• —`rope_theta` is absent from config.json; 10000.0 comes from the class default in configuration_kimi_k3.py, recorded in defaults_used rather than silently assumed.

Not included

The vision tower is excluded (MoonViT-3D, 27 blocks, plus mm_projector; 168 tensors, ~0.83 GiB). PR #26185 covers the text model only, and llama_model_loader::done_getting_tensors() runs with partial = false, so any tensor the architecture does not create makes the load fail outright. K3's MoonViT also differs from the KIMIVL tower already in clip.cpp (wqkv, patch_embed.pos_emb [64,64,1024]).

Tokenizer and chat template

K3 ships neither tokenizer.json nor tokenization_kimi.py — only encoding_k3.py and tiktoken.model. The vocabulary was rebuilt directly from tiktoken.model (sha256 identical to K2's):

• —tokenizer.ggml.model = gpt2, tokenizer.ggml.pre = kimi-k2
• —163,840 tokens, 163,328 merges, 256 single-byte tokens, 0 unused slots
• —pre-tokenizer checksum 81212dc7… matches the value llama.cpp already knows
• —BOS 163584 [BOS], EOS 163586 <|end_of_msg|> (chat-turn terminator from generation_config.json, not 163585 [EOT]/[EOS] which ends a document), EOT 163593
• —add_bos_token = false — the reference renderer does not emit BOS, so adding one would double it

A chat template IS embedded, and it was differentially tested

K3 has no static Jinja template; encoding_k3.py builds an XTML conversation in Python. Correct vocabulary alone does not reproduce the training-time format, so a Jinja equivalent is embedded as tokenizer.chat_template.

The format, read out of encoding_k3.py:

<|open|>message role="user"<|sep|>hello<|close|>message<|sep|><|end_of_msg|>
<|open|>message role="assistant"<|sep|><|open|>think<|sep|>      <- generation prompt

with _open_tag(tag, attrs) = <|open|> + tag + k="v"… + <|sep|>, _close_tag(tag) = <|close|> + tag + <|sep|>, and attribute values escaped as &→&, "→".

Differential test against the reference implementation: 28/28 cases byte-identical. The cases cover standard/multi-turn chat, CJK and attribute escaping, thinking_effort, tool declarations, assistant tool calls, ordered tool results, and nested response_schema values.

The template extracted from the published GGUF was additionally rendered by llama.cpp's Minja engine (commit 91f8c9c5). The tools/tool-result case (1,218 bytes) and nested-schema case (621 bytes) were both byte-identical to K3's own renderer. This caught and fixed three Minja-specific issues: dict-literal namespace({...}), unsupported tojson(sort_keys=true), and an items field colliding with the namespace object's method. Images and batched conversations remain untested.

Provenance and independent conformance. The file in this artifact began with Xenova's port and was expanded and patched locally; it is not claimed to be byte-identical to the separate upstream implementation in Moonshot PR #66. PR #66 and ChatLint's K3 findings provide independent, oracle-backed conformance work. The Minja namespace(items=[]) collision reproduced here was accepted and fixed upstream in ChatLint commit `ecb1a727` and on the PR branch. Against this artifact's 15,053-byte file, ChatLint pinned to that commit reports 294/294 checks, 0 errors, 0 warnings. ChatLint renders with transformers-style Jinja2 rather than Minja and checks structural properties, so it supplements rather than replaces the 28/28 oracle comparison and direct Minja runs above.

One limitation is shared with PR #66: sandboxed Jinja cannot parse a JSON string inside tool_calls[].function.arguments. This template emits a valid <|open|>json type="object" fallback for a non-empty string; callers that require the reference renderer's per-argument XTML must parse the string into an object before applying the template.

The exact same 15,053-byte template is embedded in part 1 and published separately as chat_template.jinja. Reproduce the differential test with tools/verify_chat_template.py in the conversion repository.

Files

• —94 parts, Kimi-K3-IQ1_S-000NN-of-00094.gguf. Pass part 1 to llama.cpp; the rest are discovered automatically.
• —manifest.json — per-file size and SHA-256, tensor totals, type counts.
• —quantization-report.json — error methodology, definitions, sampling coverage and gaps, distributions with bootstrap CI, and the selected operating point.
• —chat_template_verify_minja.json — the 28-case differential test against encoding_k3.py.
• —tokenizer.chat_template is embedded in part 1 (15,053 UTF-8 bytes), and the same file is available as chat_template.jinja.

License

Inherits the Kimi K3 License. Redistribution of derivative works is permitted. Commercial "Model as a Service" use above the revenue thresholds in the license requires a separate agreement with Moonshot AI.