keenanpepper/Kimi-K2.6-FP8-fused

Kimi-K2.6-FP8-fused

Kimi K2.6 in a form that plain `transformers` can actually load and run on one 8×H200 node.

This is RedHatAI/Kimi-K2.6-FP8-BLOCK (revision 317300b0ef4ec429ad7296b46e4278ab1922dd08), repacked twice with no change to any weight or scale value:

1. 1.Relabelled from the compressed-tensors block-FP8 convention to DeepSeek's finegrained_fp8 convention, which is the FP8 path transformers has real kernels for.
2. 2.Pre-fused: the 384 per-expert gate_proj / up_proj / down_proj tensors of every MoE layer are stacked offline into the fused gate_up_proj / down_proj tensors that transformers would otherwise build on the GPU during load.

Every FP8 byte and every block scale is bit-identical to RedHatAI's checkpoint, so RedHatAI's published evaluations of the quantization apply unchanged. The provenance chain is in `RELABEL.json` and `FUSE.json`, the scripts that produced it are in `tools/`, and RedHatAI's original model card is kept as `UPSTREAM_README.md`.

Why this exists

As of September 2026 there is no published Kimi K2.6 checkpoint that runs under stock transformers:

• —`moonshotai/Kimi-K2.6` is int4 compressed-tensors (~571 GB of packed experts). vLLM and SGLang serve it on 8×H200 because they have W4A16 kernels that dequantize inside the matmul. transformers has no such kernel: its only gated kernel path is FP8, so int4 falls through to the generic compressed-tensors route, whose forward pre-hook decompresses the whole model to bf16 in place on first use. That is ~1.015 T expert parameters × 2 bytes ≈ 2 TB resident, against 1128 GB of node. It OOMs, always.
• —`RedHatAI/Kimi-K2.6-FP8-BLOCK` is the right numbers in the wrong wrapper. Its 128×128 block scales are the same grid and orientation DeepSeek-V3 and GLM-5 FP8 checkpoints use, but transformers' compressed-tensors FP8 kernel (CompressedTensorsFP8Linear) is row-wise only: it allocates weight_scale as [1, out_features], a block-scaled checkpoint does not fit it, and the loader falls back to the same decompress-everything route as above.

The difference between FP8-BLOCK and a checkpoint transformers runs natively is a tensor name, a dtype, and a config block. This repo is that difference, applied.

Who this is for: anyone who needs K2.6 inside transformers rather than behind a serving engine. Interpretability work, activation and residual-stream logging, custom decoding and cache experiments, anything that hooks modules directly. If you only want tokens out, use vLLM or SGLang with the original int4 or with FP8-BLOCK; they are faster and better tested for that.

What changed, exactly

Step 1: relabel (tools/fp8_relabel.py)

69 486 scale tensors renamed and recast. No FP8 weight byte was read or written. The skip list was checked on the meta device: the set of modules the list would quantize equals the set that carries a scale on disk.

Step 2: fuse (tools/fp8_fuse_experts.py)

transformers declares this conversion for the model and applies it at load time:

source: ['mlp.experts.*.gate_proj.weight', 'mlp.experts.*.up_proj.weight']
target: ['mlp.experts.gate_up_proj']
ops   : [MergeModulelist(dim=0), Concatenate(dim=1)]

On K2.6 that stack is 384 × 2048 × 7168 FP8 = 5.25 GiB per half, built on whichever GPU the layer landed on. With 61 layers over 8 cards, at least five cards hold 8 layers (~127 GiB resident) and have ~12 GiB left to absorb it. Measured allocator fragmentation on an 8×H200 node came in at 6.70–7.27 GiB against a 6.68 GiB bar, so the un-fused load succeeded roughly one attempt in six, with byte-identical settings.

MergeModulelist passes an already-stacked tensor straight through, so a checkpoint that ships the fused tensor never pays the transient. This step does that stack and concatenation once, offline, on CPU, in the exact layout transformers' FP8Experts allocates:

92 fused tensors written from 52 992 per-expert tensors, across the 60 expert-bearing layers (layer 0 is dense). Every input byte appears in the output exactly once. Shapes were asserted against the config, not assumed.

Note the fused scale name: gate_up_proj_scale_inv, a sibling parameter, not gate_up_proj.weight_scale_inv. FP8Experts holds the projection as a bare Parameter, not a Linear, and a wrongly named scale key does not raise: the load completes with the checkpoint key reported UNEXPECTED, the parameter MISSING, and the scales left as uninitialised memory. This repo's keys were verified by reading every fused scale back after the fix (see FUSE.json, repaired). If you ever re-derive this checkpoint, read the load report.

Loading

Tested with transformers==5.15.0, torch==2.13.0+cu130, 8×H200 (141 GB), bf16 activations. Loads in about 7 minutes and has come up first try every time.

import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"  # set before torch imports

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer

path = "keenanpepper/Kimi-K2.6-FP8-fused"

model = AutoModelForImageTextToText.from_pretrained(
    path,
    trust_remote_code=False,           # native kimi_k25 class; no repo code needed
    dtype=torch.bfloat16,
    device_map="auto",
    max_memory={i: "128GiB" for i in range(8)},
    attn_implementation="eager",
)
model.eval()

# Tokenizer: use the repo's TikTokenTokenizer, NOT the converted fast tokenizer
# (see "Tokenizer" below for why).
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

Things in that snippet that are pinned rather than chosen:

• —`max_memory` of 128 GiB per card, from both sides. 61 layers over 8 cards means at least five cards must hold 8 layers; the largest layer is 15.91 GiB, so the cap must admit ≥127.3 GiB. A cap in [126, 127] GiB admits only 7 layers per card (56 < 61) and device_map="auto" then silently spills the rest to CPU or disk. Do not derive the cap from the average layer size. Print model.hf_device_map and check that nothing landed on cpu or disk.
• —`expandable_segments:True`. Without it, measured allocator fragmentation during load of the un-fused checkpoint tripled (~21 GiB). It has not been re-measured without the setting on the fused checkpoint, so keep it. Set it in the environment before torch is imported.
• —Multi-GPU in one process routes FP8 through Triton / `grouped_mm`, not DeepGEMM. transformers logs this at load. DeepGEMM's cached kernels are bound to one CUDA context; the Triton path is correct across devices, just slower.
• —`AutoModelForCausalLM` does not map `kimi_k25`. Use AutoModelForImageTextToText or the class directly.

Steady state after load is comfortable: roughly 20 GiB free per card with the weights resident, which is plenty of room for KV cache and activations at ordinary context lengths.

Tokenizer

The checkpoint ships a tiktoken BPE vocabulary (tiktoken.model) and Moonshot's tokenization_kimi.py. With trust_remote_code=False, transformers converts the vocabulary into a fast tokenizer, and that conversion renumbers 7 of the 12 added control tokens, packing them contiguously and skipping the id gaps the checkpoint leaves:

The converted tokenizer is self-consistent (it round-trips its own ids), so nothing raises and the model still produces coherent text, because <|im_end|> (163586) happens to survive. But the chat template is then feeding the model <|media_content|> where it should see <think>. The model itself settles which numbering is correct: asked to think, it emits 163607 to close its reasoning. Load the tokenizer with trust_remote_code=True so the repo's TikTokenTokenizer is used, and the ids match the embedding table.

transformers also warns that the tiktoken-converted pre-tokenizer regex may differ from the original (fix_mistral_regex). That warning was not investigated for this release; it concerns the fast-tokenizer conversion, which the recommendation above avoids.

Verification, and its limits

Verified with an engine-vs-reference harness that runs a separate implementation of the same architecture against transformers' own forward on these weights, on 8×H200 in bf16:

The independent engine sits inside transformers' own self-disagreement between its two code paths, and the residual differences are what bf16 activations over 61 layers of E4M3 weights predict (a 16B unquantized bf16 control on the same kernels shows ~2 % relative; fp32 on the same code shows ~1e-6). A 1 % relative tolerance is below the floor for a model of this depth in this precision; 5 % passes.

What this does not establish:

• —Nothing here was compared against the int4 `moonshotai/Kimi-K2.6` original. No reference of it fits on one node under transformers. The quantization quality claim rests entirely on RedHatAI's evaluations of FP8-BLOCK, which transfer because the bytes are identical.
• —The vision tower is untested. It is carried through unchanged (it is in modules_to_not_convert, so it is bf16), and the processor files are included, but only the text path has been exercised.
• —Only greedy decoding and logit agreement were checked, not benchmarks.

Caveats

• —This layout is coupled to `transformers` 5.x internals. The fused tensor names and shapes are whatever FP8Experts in integrations/finegrained_fp8.py allocates in 5.15. If a later release changes that layout or the sibling _scale_inv naming, this checkpoint will load with MISSING and UNEXPECTED keys rather than fail loudly. Check the load report on any new transformers version. The un-fused relabelled form (step 1 alone) is more portable and is ~30 minutes to regenerate (see below).
• —RedHatAI's card describes FP8-BLOCK as preliminary. This repo is pinned to revision 317300b0ef4ec429ad7296b46e4278ab1922dd08 and will not track upstream changes.
• —Do not attempt this on fewer than 8×H200-class GPUs without expecting offload. Resident weights are ~1 TB.

Reproducing it

Both scripts are in `tools/` and depend only on torch, safetensors, transformers (for --check in the relabel step) and huggingface_hub. Measured wall-clock on a 16-core CPU job with fast storage:

Each step needs ~1 TB of scratch for its output; neither holds more than one shard in memory. The relabel step refuses to write unless its derived skip list matches the modules that actually carry scales on disk; the fuse step refuses if any layer's experts are split across shards or the geometry disagrees with the config.

Files

Licence and credits

• —Weights: Moonshot AI, under the Modified MIT License (MIT plus an attribution requirement for very large commercial deployments). Redistributed with the licence and notices intact.
• —FP8 quantization: RedHatAI, via LLM Compressor. Their evaluations are in UPSTREAM_README.md.
• —Relabel and fuse: Keenan Pepper (AE Studio), as part of the Serger project. The conversion scripts in tools/ are released under the MIT licence.

If you find a problem with this repack specifically (as opposed to the model or the quantization), open a discussion on this repo.