Modelpublic

catalystsec/Kimi-K3-Audit-240E-MLX-MXFP4

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes1kdownloads

Model Card

Kimi-K3-Audit-240E-MLX-MXFP4

Kimi-K3-Audit-240E is Kimi K3 specialised for security auditing. It keeps 240 of the 896 routed experts in each layer, which brings the model onto a single 512 GB Apple silicon machine. Expert selection was calibrated on kernel source and audit traces, targeting domain focus rather than compression.

For a pair of 512 GB machines, the larger Kimi-K3-Audit-451E build measures at parity with the full release.


Base	`moonshotai/Kimi-K3`, MXFP4 release
Routed experts	240 of 896 per layer
Retained expert weights	bit-identical to the release
Size	408 GiB, 91 shards
Modality	text, vision tower is retained but unevaluated
Loader	`mlx-lm` with Kimi K3 support (PR #1626)

Intended use

Security auditing and analysis of source code, with tool use. General conversation, translation and non-technical writing are out of scope, and a general-purpose model should be used for them.

corpus	perplexity ratio	accuracy 896E	accuracy 240E	change, pts
XNU kernel C	1.07×	80.8%	79.1%	−1.7
Linux kernel C	1.05×	92.3%	90.8%	−1.5
English prose	19.0×	92.0%	38.5%	−53.5

Across eight code corpora in five languages, pruning changes perplexity least on C and C++, ranging from 1.05× on Linux to 1.38× on Python.

[image]

Evaluation

Measured against the original MXFP4 release (896E) on identical tokens. KLD is KL(896E ‖ 240E) over the full output distribution.

Audit traces

Each of 24 kernel-audit sessions contributes one 1025-token window, rendered through the model's own chat template so the structure matches deployment.

span	ppl 896E	ppl 240E	mean KLD	median KLD	top-1 agreement
tool call	1.616	1.635	0.068	0.00001	95.5%
think	5.874	6.796	0.205	0.119	78.1%
response	3.728	3.554	0.229	0.018	87.4%
other	5.044	5.880	0.467	0.254	70.3%
all	4.232	4.419	0.236	0.057	83.0%

Tool-call spans diverge least of any span measured. 78% of tool-call tokens fall below 0.01 divergence, against 16% of thinking tokens.

[image]

Code and prose corpora

corpus	language	ppl 896E	ppl 240E	mean KLD	top-1 agreement
Linux kernel	C	1.374	1.448	0.076	95.2%
XNU	C	2.213	2.361	0.089	91.6%
Swift stdlib	Swift	1.546	1.702	0.118	92.7%
llama.cpp	C++	1.664	1.844	0.142	92.0%
JavaScriptCore	C++	1.560	1.820	0.194	90.6%
V8	C++	1.660	1.979	0.195	90.0%
Linux rust/	Rust	1.596	2.028	0.266	87.6%
SGLang	Python	1.719	2.367	0.354	85.5%
wikitext	English	1.402	26.632	2.934	40.2%

Build comparison

build	size	perplexity ratio	512 GB machines
MXFP4 release, 896E	1.56 TB	1.00×	4
2bit-UVMAX, 896E	761 GiB	1.20×	2
451E	725 GiB	0.99×	2
240E, this model	408 GiB	1.04×	1

Usage

This model requires Kimi K3 support from mlx-lm PR #1626, which has not yet been merged. Until it lands in an mlx-lm release, install from the PR branch.

bash

pip install git+https://github.com/ml-explore/mlx-lm.git@refs/pull/1626/head
pip install tiktoken

python

from mlx_lm import load, generate

model, tokenizer = load(
    "catalystsec/Kimi-K3-Audit-240E-MLX-MXFP4",
    tokenizer_config={"trust_remote_code": True},
    trust_remote_code=True,
)

prompt = "Audit do_wp_page() in mm/memory.c for memory-safety issues."
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
response = generate(model, tokenizer, prompt=prompt, verbose=True)

Prefill step size

Prefill memory scales with chunk × context and is separate from the weights, so long contexts need a smaller step size than the default. Maximum context on a 512 GB M3 Ultra at batch size 1:

`--prefill-step-size`	max context
2048, the default	~198k
512	~528k
256	~731k

bash

mlx_lm.server --model catalystsec/Kimi-K3-Audit-240E-MLX-MXFP4 \
              --prefill-step-size 512

Throughput

Batch size 1 on 512 GB machines at the default step size. The two-machine figures use tensor parallelism over Thunderbolt RDMA, with peak memory per node.

1× M3 Ultra

prompt tokens	prefill tok/s	generation tok/s	peak memory
512	80.0	7.3	411 GiB
8k	91.1	7.2	422 GiB
16k	88.0	7.0	426 GiB
32k	81.8	6.9	433 GiB
64k	71.6	6.4	447 GiB

2× M3 Ultra

prompt tokens	prefill tok/s	generation tok/s	peak memory
512	138.7	11.0	211 GiB
8k	161.2	10.8	215 GiB
16k	156.7	10.7	217 GiB
32k	146.9	10.3	221 GiB
64k	129.5	9.6	228 GiB

Generation is bandwidth-bound and changes little with context, since the latent KV cache is small next to the weights read per token. A second machine gives 1.5× generation and 1.8× prefill.

Sampling follows the base model, temperature 1.0 and top-p 0.95.

Limitations

Evaluation is teacher-forced throughout and measures next-token prediction on reference text. Generation-time behaviour, including loop rate and stop-token reliability, was not evaluated.

Standard downstream benchmarks were not evaluated. Pruning preserves next-token performance less well on Rust and Python than on C, C++ and Swift.

License

Inherits the Kimi K3 license from the base model.