Modelpublic

huawei-csl/Kimi-Linear-48B-A3B-Instruct-3bit-SINQ

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face

1likes34downloads

Model Card

🐙 <a href="https://github.com/huawei-csl/SINQ">Github</a>   |   📄 <a href="https://huggingface.co/papers/2509.22944">Paper</a>

SINQ 3-bit Quantized Kimi-Linear-48B-A3B-Instruct model

This repository contains the official 3-bit quantized version of the `Kimi-Linear-48B` model using the SINQ (Sinkhorn-Normalized Quantization) method. SINQ is a novel, fast and high-quality quantization method designed to make any Large Language Models smaller while keeping their accuracy almost intact.

The method was presented in the paper SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights.

Authors: Lorenz K. Müller, Philippe Bich, Jiawei Zhuang, Ahmet Çelik, Luca Benfenati, Lukas Cavigelli.

To support the project please put a star ⭐ in the official SINQ github repository.

Model Details

—Model Name: Kimi-Linear-48B-A3B-Instruct-3bit-SINQ
—Base Model: `Kimi-Linear-48B`
—Task: Text Generation
—Framework: PyTorch / Transformers
—License: Apache-2.0
—Quantized By: Huawei - Computing Systems Lab

Quantization Details

—Quantization Method: SINQ (Sinkhorn-Normalized Quantization)
—Precision: INT3
—Group Size: 64
—Framework: PyTorch
—Quantization Library: sinq

🚀 Usage

Prerequisite

Before running the quantization script, make sure the SINQ library is installed. Installation instructions and setup details are available in the SINQ official github repository.

Usage example

You can load and use the model with our wrapper based on the 🤗 Transformers library:

python

import torch
from transformers import AutoTokenizer
from sinq.patch_model import AutoSINQHFModel

model_name = "huawei-csl/Kimi-Linear-48B-A3B-Instruct-3bit-SINQ"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
device = "cuda:0"

sinq_model = AutoSINQHFModel.from_quantized_safetensors(
    model_name,
    device=device,
    attn_implementation = "kernels-community/flash-attn2",
    compute_dtype=torch.bfloat16,
    trust_remote_code=True
)

# Test the quantized model
messages = [
    {"role": "system", "content": "You are a helpful assistant provided by Moonshot-AI."},
    {"role": "user", "content": "Is 7 a prime?"}
]

chat_prompt = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,  # ask it to end with assistant turn
)
inputs = tokenizer(
    chat_prompt,
    return_tensors="pt"
)

inputs = {k: v.to(sinq_model.device) for k, v in inputs.items()}
if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token

with torch.no_grad():
    generated_ids = sinq_model.generate(
        input_ids=inputs["input_ids"],
        attention_mask=inputs.get("attention_mask", None),
        max_new_tokens=200
    )

new_tokens = generated_ids[0, inputs["input_ids"].shape[-1]:]
response = tokenizer.decode(new_tokens, skip_special_tokens=True)
print(response)

<details> <summary>🧩 Quantization Process</summary>

The quantized model was obtained using the SINQ quantization library, following the steps below:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from sinq.patch_model import AutoSINQHFModel
from sinq.sinqlinear import BaseQuantizeConfig

# Load base model
model_name = "moonshotai/Kimi-Linear-48B-A3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    trust_remote_code=True,
    attn_implementation = "kernels-community/flash-attn2",
)

# Apply 3-bit SINQ quantization
quant_cfg = BaseQuantizeConfig(
    nbits=3,            # quantization bit-width
    group_size=64,     # group size
    tiling_mode="1D",   # tiling strategy
    method="sinq"       # quantization method ("asinq" for the calibrated version)
)

sinq_model = AutoSINQHFModel.quantize_model(
    model,
    tokenizer=tokenizer,
    quant_config=quant_cfg,
    compute_dtype=torch.bfloat16,
    device="cuda:0"
)

Reproducibility Note: This model was quantized using the SINQ implementation from commit `ee1dc76` of the SINQ repository.

</details>

🧾 How to Cite This Work

If you find SINQ useful in your research or applications, please

—Put a star ⭐ in the official SINQ github repository.
—Cite our <a href="https://huggingface.co/papers/2509.22944" target="_blank">paper</a>:

bibtex

@misc{muller2025sinq,
      title={SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights}, 
      author={Lorenz K. Muller and Philippe Bich and Jiawei Zhuang and Ahmet Celik and Luca Benfenati and Lukas Cavigelli},
      year={2025},
      eprint={2509.22944},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={http://arxiv.org/abs/2509.22944}
}