gratex/Kimi-K2.6-REAM-25

Kimi K2.6 REAM-25% — INT4 Quantization

This is a 25% expert-merged, INT4 quantized variant of moonshotai/Kimi-K2.6, a ~1T parameter Mixture-of-Experts (MoE) multimodal model with native vision capabilities.

Why this model exists: The full Kimi-K2.6 model (384 experts per MoE layer, ~555 GB) cannot fit on 4× H200 (4× 141 GB = 564 GB) with any meaningful KV cache, making the full 262,144-token context window unusable. This variant merges 25% of routed experts via REAM and quantizes to INT4, reducing model size to ~422 GB — fitting on 4× H200 with sufficient VRAM headroom for KV cache at full 256K context. All throughput benchmarks below were measured on 4× H200 NVL.

If you have access to larger hardware (8× H200, B200, etc.), you are better off using the [full moonshotai/Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) model instead. This quantized variant is specifically for 4× H200 deployments where the full model cannot run.

The base model has 384 routed experts per MoE layer. REAM (Router-weighted Expert Activation Pruning) reduces this to 288 experts (25% merge) based on saliency scores from a multi-domain calibration dataset. The resulting model is quantized to INT4 g32 symmetric pack-quantized format (compressed-tensors) with full vision encoder preserved in BF16.

Model Details

Quantization Details

Quantization Configuration

{
  "config_groups": {
    "group_0": {
      "targets": ["Linear"],
      "weights": {
        "num_bits": 4,
        "type": "int",
        "group_size": 32,
        "strategy": "group",
        "symmetric": true
      }
    }
  },
  "format": "pack-quantized",
  "quant_method": "compressed-tensors",
  "quantization_status": "compressed"
}

Key parameters:

• —INT4 symmetric: 4-bit integer weights with symmetric quantization (no zero-point)
• —group_size=32: Per-32-element scaling groups (fine-grained, high quality)
• —pack-quantized format: Packed INT4 weights for efficient storage and Marlin kernel execution
• —Expert merging via REAM: Router-weighted saliency scores from a multi-domain calibration dataset determine which experts to merge; 25% of routed experts are pruned by fusing low-saliency experts into remaining ones

Calibration Dataset

The REAM expert merging and INT4 quantization calibration used a multi-domain composite dataset of ~1,280 batches (batchsize=4, ~5,120 samples, modelmax_length=2048):

Quality Benchmarks

All benchmarks on wikitext-2-raw-v1 (test split), 512-token chunks.

WikiText-2 Perplexity (ctx=512)

Throughput Benchmarks

All throughput benchmarks measured on 4× H200 NVL (4× 141 GB, tensor-parallel=4, expert-parallel enabled).

Single Request (concurrency=1)

20 requests, max_tokens=1024, temperature=0.9.

16 Concurrent Requests

320 total requests, max_tokens=1024, temperature=0.9.

Hardware Requirements

Minimum: 4× GPU with ≥120 GB total VRAM for this quantized model.

Users with 8× H200, B200, or larger setups: the full unquantized moonshotai/Kimi-K2.6 will deliver better quality. This model exists because the full model cannot run on 4× H200 with usable KV cache.

INT4 pack-quantized requires CUDA GPU sm80+ (Ampere and newer). This model uses custom MoE linear layers with INT4 weights + BF16 activations.

When to Use This Model

This model exists because the full Kimi-K2.6 (~555 GB, 384 experts) cannot run on 4× H200 with any usable KV cache. REAM expert merging + INT4 quantization reduces the model to ~422 GB, leaving enough VRAM for KV cache at full 262,144-token context. If your hardware can accommodate the full model, use it — quality will be higher.

Usage with vLLM

Tested with: vllm/vllm-openai:v0.20.0-cu130-ubuntu2404 on 4× H200 NVL

Docker Deployment

docker run -d --name vllm-kimi \
  --gpus all \
  --shm-size=16g \
  --network host \
  --ipc host \
  --pid host \
  --restart=unless-stopped \
  -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
  -e NCCL_P2P_DISABLE=0 \
  -e NCCL_P2P_LEVEL=NVL \
  -e NCCL_IB_DISABLE=1 \
  -e NCCL_NET_DISABLE=1 \
  -e NCCL_SHM_DISABLE=0 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -v /path/to/model:/model \
  vllm/vllm-openai:v0.20.0-cu130-ubuntu2404 \
    /model \
    --served-model-name kimi-k2.6-ream-25 \
    --trust-remote-code \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --disable-custom-all-reduce \
    --gpu-memory-utilization 0.92 \
    --max-model-len 262144 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 8 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --kv-cache-dtype fp8_e4m3 \
    --tool-call-parser kimi_k2 \
    --reasoning-parser kimi_k2 \
    --enable-auto-tool-choice \
    --dtype bfloat16 \
    --host 0.0.0.0 \
    --port 8000

Example vLLM Configuration (YAML)

This configuration is deployed and verified on 4× H200 NVL:

# -- Model & Server ----------------------------------------------------------
model: /model
host: "0.0.0.0"
port: 8000
served_model_name: "kimi-k2.6-ream-25"
trust_remote_code: true
tensor_parallel_size: 4
enable_expert_parallel: true

# -- Quantization ------------------------------------------------------------
quantization: compressed-tensors

# -- Data Type ---------------------------------------------------------------
dtype: bfloat16

# -- Context & Batching ------------------------------------------------------
max_model_len: 262144
max_num_batched_tokens: 8192
max_num_seqs: 8
enable_chunked_prefill: true

# -- Memory ------------------------------------------------------------------
gpu_memory_utilization: 0.92
enable_prefix_caching: true
kv_cache_dtype: fp8_e4m3

# -- Tool Calling & Reasoning ------------------------------------------------
enable_auto_tool_choice: true
tool_call_parser: kimi_k2
reasoning_parser: kimi_k2

# -- NCCL / Multi-GPU --------------------------------------------------------
disable_custom_all_reduce: true

# -- Misc --------------------------------------------------------------------
enforce_eager: false

Inference Test

# Text completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"kimi-k2.6-ream-25","messages":[{"role":"user","content":"What is 2+2? One word."}],"max_tokens":10}'

Transformers / Python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "gratex/Kimi-K2.6-REAM-25"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

messages = [{"role": "user", "content": "What is 2+2?"}]
text = tokenizer.apply_chat_template(messages, return_tensors=True)
output = model.generate(**text, max_new_tokens=256)

Files in This Repository

License

This quantized model is released under the Modified MIT License, following the base model's license.

The base model moonshotai/Kimi-K2.6 is licensed under Modified MIT. See `LICENSE` for the full license text.

Citation

If you use this model in your research or production systems, please cite:

@misc{kimi-k2.6-ream-25,
  title  = {Kimi K2.6 REAM-25% INT4 Quantization},
  author = {Gratex International},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/gratex/Kimi-K2.6-REAM-25}},
  note   = {REAM expert merge + INT4 quantization, 4× H200 deployment}
}

Acknowledgments

This quantization was produced using hardware and infrastructure provided by Gratex International, a.s.

Base Model: moonshotai/Kimi-K2.6 Expert Merging: REAM (Router-weighted Expert Activation Pruning) Quantization Tool: llm-compressor Quantization Format: compressed-tensors Deployment Engine: vLLM