nota-ai/Kimi-K3-Nota-Global-Pruned-25

Nota Global-Pruned Kimi K3 (25%)

Highlight

Nota's global expert pruning is applied: saliency is measured on a global scale so that each layer keeps only the experts that matter to it. The result is a non-uniform checkpoint — every layer holds a different number of experts, 379 to 896 across the 92 MoE layers, 61,824 in total.

• —The per-layer counts ship in config.json as n_routed_experts_per_layer, and the vLLM modeling file is patched to read it. Stock vLLM sizes every layer from the single scalar num_experts and cannot serve a non-uniform checkpoint.
• —25% pruning drops the deployment from B300 × 8 to B300 × 6.

Performance

Better than REAP (ICLR 2026), the method behind the pruned Kimi K3 variants published on Hugging Face.

At 25% the pruned model matches or beats the unpruned Kimi K3 on every benchmark while running on two fewer GPUs.

Evaluation config, per task:

• —IFEval — thinking on, max_tokens=32768.
• —GPQA-Diamond — thinking on, max_tokens=32768.
• —IFBench — thinking on, max_tokens=32768.
• —HumanEval+ — thinking on, max_tokens=32768.

More benchmarks will be added to this table.

Installation

Built against the vLLM version from the official Kimi K3 recipe, installed as documented there (as of 2026-07-27):

vllm 0.1.dev19262+gb6bbf29dd.d20260727

Replace model.py in that install with patch/model.py:

cp /workspace/kimi-k3/patch/model.py \
   /path/to/vllm/models/kimi_k3/nvidia/model.py

Find the exact path with:

python3 -c "import vllm, pathlib; print(pathlib.Path(vllm.__file__).parent / 'models/kimi_k3/nvidia/model.py')"

model.py.orig is the stock file, kept for restoring.

Quick Start

B300 × 6

VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 \
FLASHINFER_PRIVATE_CUBIN_DIR=/opt/flashinfer-local-cubins/20260617_v0613rc1_situ_v0611_barrierfix \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 \
vllm serve /workspace/kimi-k3/models/global_pruned_25_kimi-k3 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --moe-backend auto \
  --gpu-memory-utilization 0.95 \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 3 \
  --max-num-seqs 256 \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

Six GPUs are reached as TP=2 × PP=3, not TP=6: vocab_size is 163840 = 2¹⁵ × 5 and MoonViT's vt_intermediate_size is 4096, so TP must be a power of two.

Do not add --enable-expert-parallel — EP requires num_experts % ep_size == 0, which a non-uniform checkpoint cannot satisfy.

See /workspace/kimi-k3/inference/run_global_25.sh for the same launch with a pre-flight memory check.