AaryanK/Kimi-Linear-48B-A3B-Instruct-GGUF

Kimi-Linear-48B-Instruct-GGUF

<div align="center"> <img src="https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/resolve/main/figures/logo.png" width="5%" style="vertical-align: middle;"/> <b>Kimi Linear: An Expressive, Efficient Attention Architecture</b> </div>

<div align="center">   <a href="https://huggingface.co/papers/2510.26692" style="margin: 0 8px;"> <b>Paper</b> </a>   <a href="https://github.com/MoonshotAI/Kimi-Linear" style="margin: 0 8px;"> <b>Code</b> </a>   <a href="https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct" style="margin: 0 8px;"> <b>Original Model</b> </a> </div>

<br>

[!IMPORTANT] I am currently looking for open positions! 🤗 If you find this model useful or are looking for a talented AI/LLM Engineer, please reach out to me on LinkedIn: [Aaryan Kapoor](https://www.linkedin.com/in/theaaryankapoor/).

[!CAUTION] Experimental Build Required 🚧 This model utilizes the Kimi Delta Attention (KDA) architecture, which is not yet supported in the main branch of llama.cpp. To run this GGUF, you must compile llama.cpp from [PR #17592](https://github.com/ggml-org/llama.cpp/pull/17592). Attempting to run this on a standard build will result in errors.

<div align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/64e1a459ff3fd4fd8eedb456/IlxKYaUsl15644y-k8GNo.png" width="70%" style="vertical-align: middle;"/> <b>Some test prompts :)</b> </div>

Description

This repository contains experimental GGUF format model files for Moonshot AI's Kimi Linear 48B.

Kimi Linear is a hybrid linear attention architecture designed to outperform traditional full attention methods in long-context and scaling regimes. It uses Kimi Delta Attention (KDA) and a hybrid architecture (3:1 KDA-to-MLA ratio) to reduce memory usage and boost throughput by up to 6x on long sequences.

Performance & Architecture. This model is currently quantized to Q2_K (and others) to fit on consumer hardware while testing the architecture's correctness. Despite the aggressive quantization, initial tests show the logic and reasoning capabilities remain intact.

How to Run (llama.cpp)

Prerequisite: You must clone and build the specific PR branch:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/17592/head:pr-17592
git checkout pr-17592
make -j

1. CLI Inference (Interactive Chat)

./llama-cli -m Kimi-Linear-48B-Instruct.Q2_K.gguf \
  -n 2048 \                  # Adjust generation limit
  -c 8192 \                  # Context window (Model supports up to 1M)
  --temp 0.8 \               # Recommended temperature
  --top-p 0.9 \
  -ngl 99 \                  # Offload all layers to GPU
  -p "<|im_start|>user\nHello, who are you?<|im_end|>\n<|im_start|>assistant\n" \
  -cnv

Note: The current GGUF implementation successfully mitigates previous "state collapse" issues found in early development.

2. Server Mode (API)

Running a persistent server is recommended for this size model to avoid reloading times.

./llama-server -m Kimi-Linear-48B-Instruct.Q2_K.gguf \
  --port 8080 \
  -ngl 99 \
  -c 8192 \
  --alias kimi

Hardware Requirements

• —Full GPU Offloading (`-ngl 99`):
• —Q4_K_M: Requires ~28GB VRAM (e.g., A100, A6000, or Mac Studio M2/M3 Max).
• —Q2_K: Requires ~16-18GB VRAM (Fits on RTX 3090 / 4090).
• —Split Offloading:
• —If you have less VRAM (e.g., 12GB), use -ngl with a lower number (e.g., -ngl 20) to split layers between GPU and CPU RAM.

Default Settings

• —temperature: 0.8
• —top-p: 0.9
• —repeat-penalty: 1.05 (Optional, if repetition occurs)

CLI Example

./llama-cli -m Kimi-Linear-48B-Instruct.Q2_K.gguf \
  -c 8192 \
  --temp 0.8 \
  --top-p 0.9 \
  -p "<|im_start|>user\nWrite a Python script to calculate Fibonacci numbers.<|im_end|>\n<|im_start|>assistant\n" \
  -cnv