openbmb/MiniCPM4-8B

1---2license: apache-2.03language:4- zh5- en6pipeline_tag: text-generation7library_name: transformers8---9<div align="center">10<img src="https://github.com/OpenBMB/MiniCPM/blob/main/assets/minicpm_logo.png?raw=true" width="500em" ></img> 11</div>12 13<p align="center">14<a href="https://github.com/OpenBMB/MiniCPM/" target="_blank">GitHub Repo</a> |15<a href="https://arxiv.org/abs/2506.07900" target="_blank">Technical Report</a> |16<a href="https://mp.weixin.qq.com/s/KIhH2nCURBXuFXAtYRpuXg?poc_token=HBIsUWijxino8oJ5s6HcjcfXFRi0Xj2LJlxPYD9c">Join Us</a>17</p>18<p align="center">19👋 Contact us in <a href="https://discord.gg/3cGQn9b3YM" target="_blank">Discord</a> and <a href="https://github.com/OpenBMB/MiniCPM/blob/main/assets/wechat.jpg" target="_blank">WeChat</a>20</p>21 22## What's New23- [2025.06.06] **MiniCPM4** series are released! This model achieves ultimate efficiency improvements while maintaining optimal performance at the same scale! It can achieve over 5x generation acceleration on typical end-side chips! You can find technical report [here](https://github.com/OpenBMB/MiniCPM/tree/main/report/MiniCPM_4_Technical_Report.pdf).🔥🔥🔥24 25## MiniCPM4 Series26MiniCPM4 series are highly efficient large language models (LLMs) designed explicitly for end-side devices, which achieves this efficiency through systematic innovation in four key dimensions: model architecture, training data, training algorithms, and inference systems.27- [MiniCPM4-8B](https://huggingface.co/openbmb/MiniCPM4-8B): The flagship of MiniCPM4, with 8B parameters, trained on 8T tokens. (**<-- you are here**)28- [MiniCPM4-0.5B](https://huggingface.co/openbmb/MiniCPM4-0.5B): The small version of MiniCPM4, with 0.5B parameters, trained on 1T tokens.29- [MiniCPM4-8B-Eagle-FRSpec](https://huggingface.co/openbmb/MiniCPM4-8B-Eagle-FRSpec): Eagle head for FRSpec, accelerating speculative inference for MiniCPM4-8B.30- [MiniCPM4-8B-Eagle-FRSpec-QAT-cpmcu](https://huggingface.co/openbmb/MiniCPM4-8B-Eagle-FRSpec-QAT-cpmcu): Eagle head trained with QAT for FRSpec, efficiently integrate speculation and quantization to achieve ultra acceleration for MiniCPM4-8B.31- [MiniCPM4-8B-Eagle-vLLM](https://huggingface.co/openbmb/MiniCPM4-8B-Eagle-vLLM): Eagle head in vLLM format, accelerating speculative inference for MiniCPM4-8B.32- [MiniCPM4-8B-marlin-Eagle-vLLM](https://huggingface.co/openbmb/MiniCPM4-8B-marlin-Eagle-vLLM): Quantized Eagle head for vLLM format, accelerating speculative inference for MiniCPM4-8B.33- [BitCPM4-0.5B](https://huggingface.co/openbmb/BitCPM4-0.5B): Extreme ternary quantization applied to MiniCPM4-0.5B compresses model parameters into ternary values, achieving a 90% reduction in bit width.34- [BitCPM4-1B](https://huggingface.co/openbmb/BitCPM4-1B): Extreme ternary quantization applied to MiniCPM3-1B compresses model parameters into ternary values, achieving a 90% reduction in bit width.35- [MiniCPM4-Survey](https://huggingface.co/openbmb/MiniCPM4-Survey): Based on MiniCPM4-8B, accepts users' quiries as input and autonomously generate trustworthy, long-form survey papers.36- [MiniCPM4-MCP](https://huggingface.co/openbmb/MiniCPM4-MCP): Based on MiniCPM4-8B, accepts users' queries and available MCP tools as input and autonomously calls relevant MCP tools to satisfy users' requirements.37 38## Introduction39MiniCPM 4 is an extremely efficient edge-side large model that has undergone efficient optimization across four dimensions: model architecture, learning algorithms, training data, and inference systems, achieving ultimate efficiency improvements.40 41- 🏗️ **Efficient Model Architecture:**42  - InfLLM v2 -- Trainable Sparse Attention Mechanism: Adopts a trainable sparse attention mechanism architecture where each token only needs to compute relevance with less than 5% of tokens in 128K long text processing, significantly reducing computational overhead for long texts43 44- 🧠 **Efficient Learning Algorithms:**45  - Model Wind Tunnel 2.0 -- Efficient Predictable Scaling: Introduces scaling prediction methods for performance of downstream tasks, enabling more precise model training configuration search46  - BitCPM -- Ultimate Ternary Quantization: Compresses model parameter bit-width to 3 values, achieving 90% extreme model bit-width reduction47  - Efficient Training Engineering Optimization: Adopts FP8 low-precision computing technology combined with Multi-token Prediction training strategy48 49- 📚 **High-Quality Training Data:**50  - UltraClean -- High-quality Pre-training Data Filtering and Generation: Builds iterative data cleaning strategies based on efficient data verification, open-sourcing high-quality Chinese and English pre-training dataset [UltraFinweb](https://huggingface.co/datasets/openbmb/Ultra-FineWeb)51  - UltraChat v2 -- High-quality Supervised Fine-tuning Data Generation: Constructs large-scale high-quality supervised fine-tuning datasets covering multiple dimensions including knowledge-intensive data, reasoning-intensive data, instruction-following data, long text understanding data, and tool calling data52 53- ⚡ **Efficient Inference System:**54  - CPM.cu -- Lightweight and Efficient CUDA Inference Framework: Integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding55  - ArkInfer -- Cross-platform Deployment System: Supports efficient deployment across multiple backend environments, providing flexible cross-platform adaptation capabilities56 57## Usage58 59### Inference with [CPM.cu](https://github.com/OpenBMB/cpm.cu)60 61We recommend using [CPM.cu](https://github.com/OpenBMB/cpm.cu) for the inference of MiniCPM4. CPM.cu is a CUDA inference framework developed by OpenBMB, which integrates efficient sparse, speculative sampling, and quantization techniques, fully leveraging the efficiency advantages of MiniCPM4.62 63You can install CPM.cu by running the following command:64 65```bash66git clone https://github.com/OpenBMB/cpm.cu.git --recursive67cd cpm.cu68python3 setup.py install69```70 71MiniCPM4 natively supports context lengths of up to 32,768 tokens. To reproduce the long-text acceleration effect in the paper, we recommend using the LongRoPE factors that have been validated. Change the `rope_scaling` field in the `config.json` file as the following to enable LongRoPE.72```json73{74    ...,75    "rope_scaling": {76        "rope_type": "longrope", 77        "long_factor": [0.9977997200264581, 1.014658295992452, 1.0349680404997148, 1.059429246056193, 1.0888815016813513, 1.1243301355211495, 1.166977103606075, 1.2182568066927284, 1.2798772354275727, 1.3538666751582975, 1.4426259039919596, 1.5489853358570191, 1.6762658237220625, 1.8283407612492941, 2.0096956085876183, 2.225478927469756, 2.481536379650452, 2.784415934557119, 3.1413289096347365, 3.560047844772632, 4.048719380066383, 4.752651957515948, 5.590913044973868, 6.584005926629993, 7.7532214876576155, 9.119754865903639, 10.704443927019176, 12.524994176518703, 14.59739595363613, 16.93214476166354, 19.53823297353041, 22.417131025031697, 25.568260840911098, 28.991144156566317, 32.68408069090375, 36.65174474170465, 40.90396065611201, 45.4664008671033, 50.37147343433591, 55.6804490772103, 61.470816952306556, 67.8622707390618, 75.00516023410414, 83.11898235973767, 92.50044360202462, 103.57086856690864, 116.9492274587385, 118.16074567836519, 119.18497548708795, 120.04810876261652, 120.77352815196981, 121.38182790207875, 121.89094985353891, 122.31638758099915, 122.6714244963338, 122.9673822552567, 123.21386397019609, 123.41898278254268, 123.58957065488238, 123.73136519024158, 123.84917421274221, 123.94701903496814, 124.02825801299717, 124.09569231686116],78        "short_factor": [0.9977997200264581, 1.014658295992452, 1.0349680404997148, 1.059429246056193, 1.0888815016813513, 1.1243301355211495, 1.166977103606075, 1.2182568066927284, 1.2798772354275727, 1.3538666751582975, 1.4426259039919596, 1.5489853358570191, 1.6762658237220625, 1.8283407612492941, 2.0096956085876183, 2.225478927469756, 2.481536379650452, 2.784415934557119, 3.1413289096347365, 3.560047844772632, 4.048719380066383, 4.752651957515948, 5.590913044973868, 6.584005926629993, 7.7532214876576155, 9.119754865903639, 10.704443927019176, 12.524994176518703, 14.59739595363613, 16.93214476166354, 19.53823297353041, 22.417131025031697, 25.568260840911098, 28.991144156566317, 32.68408069090375, 36.65174474170465, 40.90396065611201, 45.4664008671033, 50.37147343433591, 55.6804490772103, 61.470816952306556, 67.8622707390618, 75.00516023410414, 83.11898235973767, 92.50044360202462, 103.57086856690864, 116.9492274587385, 118.16074567836519, 119.18497548708795, 120.04810876261652, 120.77352815196981, 121.38182790207875, 121.89094985353891, 122.31638758099915, 122.6714244963338, 122.9673822552567, 123.21386397019609, 123.41898278254268, 123.58957065488238, 123.73136519024158, 123.84917421274221, 123.94701903496814, 124.02825801299717, 124.09569231686116],79        "original_max_position_embeddings": 3276880    }81}82```83 84After modification, you can run the following command to reproduce the long-context acceleration effect (the script will automatically download the model weights from HuggingFace)85```bash86python3 tests/test_generate.py87```88 89For more details about CPM.cu, please refer to [the repo CPM.cu](https://github.com/OpenBMB/cpm.cu).90 91### Inference with Transformers92```python93from transformers import AutoModelForCausalLM, AutoTokenizer94import torch95torch.manual_seed(0)96 97path = 'openbmb/MiniCPM4-8B'98device = "cuda"99tokenizer = AutoTokenizer.from_pretrained(path)100model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map=device, trust_remote_code=True)101 102# User can directly use the chat interface103# responds, history = model.chat(tokenizer, "Write an article about Artificial Intelligence.", temperature=0.7, top_p=0.7)104# print(responds)105 106# User can also use the generate interface107messages = [108    {"role": "user", "content": "Write an article about Artificial Intelligence."},109]110prompt_text = tokenizer.apply_chat_template(111    messages,112    tokenize=False,113    add_generation_prompt=True,114)115model_inputs = tokenizer([prompt_text], return_tensors="pt").to(device)116 117model_outputs = model.generate(118    **model_inputs,119    max_new_tokens=1024,120    top_p=0.7,121    temperature=0.7122)123output_token_ids = [124    model_outputs[i][len(model_inputs[i]):] for i in range(len(model_inputs['input_ids']))125]126 127responses = tokenizer.batch_decode(output_token_ids, skip_special_tokens=True)[0]128print(responses)129```130 131MiniCPM4-8B supports `InfLLM v2`, a sparse attention mechanism designed for efficient long-sequence inference. It requires the [infllmv2_cuda_impl](https://github.com/OpenBMB/infllmv2_cuda_impl) library.132 133You can install it by running the following command:134```bash135git clone -b feature_infer https://github.com/OpenBMB/infllmv2_cuda_impl.git136cd infllmv2_cuda_impl137git submodule update --init --recursive138pip install -e . # or python setup.py install 139```140 141To enable InfLLM v2, you need to add the `sparse_config` field in `config.json`:142```json143{144    ...,145    "sparse_config": {146        "kernel_size": 32,147        "kernel_stride": 16,148        "init_blocks": 1,149        "block_size": 64,150        "window_size": 2048,151        "topk": 64,152        "use_nope": false,153        "dense_len": 8192154    }155}156```157 158These parameters control the behavior of InfLLM v2:159* `kernel_size` (default: 32): The size of semantic kernels.160* `kernel_stride` (default: 16): The stride between adjacent kernels.161* `init_blocks` (default: 1): The number of initial blocks that every query token attends to. This ensures attention to the beginning of the sequence.162* `block_size` (default: 64): The block size for key-value blocks.163* `window_size` (default: 2048): The size of the local sliding window. 164* `topk` (default: 64): The specifies that each token computes attention with only the top-k most relevant key-value blocks.165* `use_nope` (default: false): Whether to use the NOPE technique in block selection for improved performance.166* `dense_len` (default: 8192): Since Sparse Attention offers limited benefits for short sequences, the model can use standard (dense) attention for shorter texts. The model will use dense attention for sequences with a token length below `dense_len` and switch to sparse attention for sequences exceeding this length. Set this to `-1` to always use sparse attention regardless of sequence length.167 168MiniCPM4 natively supports context lengths of up to 32,768 tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques for effective handling of long texts. We have validated the model's performance on context lengths of up to 131,072 tokens by modifying the LongRoPE factor.169 170You can apply the LongRoPE factor modification by modifying the model files. Specifically, in the `config.json` file, adjust the `rope_scaling` fields.171```json172{173    ...,174    "rope_scaling": {175        "rope_type": "longrope", 176        "long_factor": [0.9977997200264581, 1.014658295992452, 1.0349680404997148, 1.059429246056193, 1.0888815016813513, 1.1243301355211495, 1.166977103606075, 1.2182568066927284, 1.2798772354275727, 1.3538666751582975, 1.4426259039919596, 1.5489853358570191, 1.6762658237220625, 1.8283407612492941, 2.0096956085876183, 2.225478927469756, 2.481536379650452, 2.784415934557119, 3.1413289096347365, 3.560047844772632, 4.048719380066383, 4.752651957515948, 5.590913044973868, 6.584005926629993, 7.7532214876576155, 9.119754865903639, 10.704443927019176, 12.524994176518703, 14.59739595363613, 16.93214476166354, 19.53823297353041, 22.417131025031697, 25.568260840911098, 28.991144156566317, 32.68408069090375, 36.65174474170465, 40.90396065611201, 45.4664008671033, 50.37147343433591, 55.6804490772103, 61.470816952306556, 67.8622707390618, 75.00516023410414, 83.11898235973767, 92.50044360202462, 103.57086856690864, 116.9492274587385, 118.16074567836519, 119.18497548708795, 120.04810876261652, 120.77352815196981, 121.38182790207875, 121.89094985353891, 122.31638758099915, 122.6714244963338, 122.9673822552567, 123.21386397019609, 123.41898278254268, 123.58957065488238, 123.73136519024158, 123.84917421274221, 123.94701903496814, 124.02825801299717, 124.09569231686116],177        "short_factor": [0.9977997200264581, 1.014658295992452, 1.0349680404997148, 1.059429246056193, 1.0888815016813513, 1.1243301355211495, 1.166977103606075, 1.2182568066927284, 1.2798772354275727, 1.3538666751582975, 1.4426259039919596, 1.5489853358570191, 1.6762658237220625, 1.8283407612492941, 2.0096956085876183, 2.225478927469756, 2.481536379650452, 2.784415934557119, 3.1413289096347365, 3.560047844772632, 4.048719380066383, 4.752651957515948, 5.590913044973868, 6.584005926629993, 7.7532214876576155, 9.119754865903639, 10.704443927019176, 12.524994176518703, 14.59739595363613, 16.93214476166354, 19.53823297353041, 22.417131025031697, 25.568260840911098, 28.991144156566317, 32.68408069090375, 36.65174474170465, 40.90396065611201, 45.4664008671033, 50.37147343433591, 55.6804490772103, 61.470816952306556, 67.8622707390618, 75.00516023410414, 83.11898235973767, 92.50044360202462, 103.57086856690864, 116.9492274587385, 118.16074567836519, 119.18497548708795, 120.04810876261652, 120.77352815196981, 121.38182790207875, 121.89094985353891, 122.31638758099915, 122.6714244963338, 122.9673822552567, 123.21386397019609, 123.41898278254268, 123.58957065488238, 123.73136519024158, 123.84917421274221, 123.94701903496814, 124.02825801299717, 124.09569231686116],178        "original_max_position_embeddings": 32768179    }180}181```182 183### Inference with [SGLang](https://github.com/sgl-project/sglang)184 185For now, you need to install our forked version of SGLang.186```bash187git clone -b openbmb https://github.com/OpenBMB/sglang.git188cd sglang189 190pip install --upgrade pip191pip install -e "python[all]"192```193 194You can start the inference server by running the following command:195```bash196python -m sglang.launch_server --model openbmb/MiniCPM4-8B --trust-remote-code --port 30000 --chat-template chatml197```198 199Then you can use the chat interface by running the following command:200```python201import openai202 203client = openai.Client(base_url=f"http://localhost:30000/v1", api_key="None")204 205response = client.chat.completions.create(206    model="openbmb/MiniCPM4-8B",207    messages=[208        {"role": "user", "content": "Write an article about Artificial Intelligence."},209    ],210    temperature=0.7,211    max_tokens=1024,212)213 214print(response.choices[0].message.content)215```216 217### Inference with [vLLM](https://github.com/vllm-project/vllm)218For now, you need to install the latest version of vLLM.219```220pip install -U vllm \221    --pre \222    --extra-index-url https://wheels.vllm.ai/nightly223```224 225Then you can inference MiniCPM4-8B with vLLM:226```python227from transformers import AutoTokenizer228from vllm import LLM, SamplingParams229 230model_name = "openbmb/MiniCPM4-8B"231prompt = [{"role": "user", "content": "Please recommend 5 tourist attractions in Beijing. "}]232 233tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)234input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)235 236llm = LLM(237    model=model_name,238    trust_remote_code=True,239    max_num_batched_tokens=32768, 240    dtype="bfloat16", 241    gpu_memory_utilization=0.8, 242)243sampling_params = SamplingParams(top_p=0.7, temperature=0.7, max_tokens=1024, repetition_penalty=1.02)244 245outputs = llm.generate(prompts=input_text, sampling_params=sampling_params)246 247print(outputs[0].outputs[0].text)248```249 250Also, you can start the inference server by running the following command:251> **Note**: In vLLM's chat API, `add_special_tokens` is `False` by default. This means important special tokens—such as the beginning-of-sequence (BOS) token—will not be added automatically. To ensure the input prompt is correctly formatted for the model, you should explicitly set `extra_body={"add_special_tokens": True}`.252 253```bash254vllm serve openbmb/MiniCPM4-8B 255```256 257Then you can use the chat interface by running the following code:258 259```python260import openai261 262client = openai.Client(base_url="http://localhost:8000/v1", api_key="EMPTY")263 264response = client.chat.completions.create(265    model="openbmb/MiniCPM4-8B",266    messages=[267        {"role": "user", "content": "Write an article about Artificial Intelligence."},268    ],269    temperature=0.7,270    max_tokens=1024,271    extra_body=dict(add_special_tokens=True),  # Ensures special tokens are added for chat template272    273)274 275print(response.choices[0].message.content)276```277 278## Evaluation Results279On two typical end-side chips, Jetson AGX Orin and RTX 4090, MiniCPM4 demonstrates significantly faster processing speed compared to similar-size models in long text processing tasks. As text length increases, MiniCPM4's efficiency advantage becomes more pronounced. On the Jetson AGX Orin platform, compared to Qwen3-8B, MiniCPM4 achieves approximately 7x decoding speed improvement.280 281![benchmark](https://github.com/OpenBMB/MiniCPM/blob/main/assets/minicpm4/efficiency.png?raw=true)282 283#### Comprehensive Evaluation284MiniCPM4 launches end-side versions with 8B and 0.5B parameter scales, both achieving best-in-class performance in their respective categories.285 286![benchmark](https://github.com/OpenBMB/MiniCPM/blob/main/assets/minicpm4/benchmark.png?raw=true)287 288#### Long Text Evaluation289MiniCPM4 is pre-trained on 32K long texts and achieves length extension through YaRN technology. In the 128K long text needle-in-a-haystack task, MiniCPM4 demonstrates outstanding performance.290 291![long-niah](https://github.com/OpenBMB/MiniCPM/blob/main/assets/minicpm4/128k-niah.png?raw=true)292 293## Statement294- As a language model, MiniCPM generates content by learning from a vast amount of text. 295- However, it does not possess the ability to comprehend or express personal opinions or value judgments. 296- Any content generated by MiniCPM does not represent the viewpoints or positions of the model developers. 297- Therefore, when using content generated by MiniCPM, users should take full responsibility for evaluating and verifying it on their own.298 299## LICENSE300- This repository and MiniCPM models are released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) License. 301 302## Citation303- Please cite our [paper](https://github.com/OpenBMB/MiniCPM/tree/main/report/MiniCPM_4_Technical_Report.pdf) if you find our work valuable.304 305```bibtex306@article{minicpm4,307  title={{MiniCPM4}: Ultra-Efficient LLMs on End Devices},308  author={MiniCPM Team},309  year={2025}310}311```