CoolFace
Modelpublic

NANI-Nithin/MiniCPM5-2B-GGUF

sourceHugging Faceapache-2.0updated 15d agoView on Hugging Face
8likes15kdownloads
Model Card

MiniCPM5-2B — GGUF quants

Full llama.cpp quant sweep of openbmb/MiniCPM5-2B, OpenBMB's dense 2B model for on-device and local deployment. 30 files — the BF16 source plus 29 quants from Q80 down to Q10 — all cut from the official BF16 release, with the importance matrix computed on the BF16 weights.

The source model uses the standard LlamaForCausalLM architecture — no custom kernels and no model-code fork are needed; these files load directly in llama.cpp, Ollama and LM Studio. (OpenBMB also publishes their own MiniCPM5-2B-GGUF; this repo complements it with the complete quant sweep including the IQ and extreme quants. For vLLM, SGLang or transformers, use the source repo directly.)

Which file should I get?

FileSizeUse
MiniCPM5-2B-BF16.gguf4.69 GBFull precision source. Every quant below is cut from this file.
MiniCPM5-2B-Q8_0.gguf2.50 GBEffectively lossless. Use when disk and RAM are not the constraint.
MiniCPM5-2B-Q6_K.gguf1.93 GBNear-lossless; the last stop before quality becomes measurable.
MiniCPM5-2B-Q5_1.gguf1.78 GBLegacy. Prefer Q5KM.
MiniCPM5-2B-Q5KM.gguf1.68 GBVery good quality, noticeably smaller than Q6_K.
MiniCPM5-2B-Q5KS.gguf1.65 GBSlightly smaller than Q5KM for a slight quality cost.
MiniCPM5-2B-Q5_0.gguf1.65 GBLegacy. Prefer Q5KM.
MiniCPM5-2B-Q4_1.gguf1.52 GBLegacy. Prefer Q4KM.
MiniCPM5-2B-Q4KM.gguf1.45 GBThe usual default. Best quality-per-byte for most people.
MiniCPM5-2B-Q4KS.gguf1.40 GBA little smaller than Q4KM, a little worse.
MiniCPM5-2B-IQ4_NL.gguf1.39 GBNon-linear 4-bit; good on hardware without fast K-quant kernels.
MiniCPM5-2B-Q4_0.gguf1.39 GBLegacy round-to-nearest. Prefer Q4KM unless a runtime needs this.
MiniCPM5-2B-IQ4_XS.gguf1.33 GBBest value at this size; usually beats Q4KS while smaller.
MiniCPM5-2B-Q3KL.gguf1.28 GBSmall, with real quality loss. Usable when RAM is tight.
MiniCPM5-2B-Q3KM.gguf1.20 GBSmaller again; noticeable degradation.
MiniCPM5-2B-IQ3_M.gguf1.14 GBStrong 3-bit choice, clearly better than Q3KM.
MiniCPM5-2B-IQ3_S.gguf1.11 GBSlightly smaller than IQ3_M.
MiniCPM5-2B-Q3KS.gguf1.11 GBAggressive. Prefer IQ3_M at a similar size.
MiniCPM5-2B-IQ3_XS.gguf1.07 GBAggressive but coherent.
MiniCPM5-2B-IQ3_XXS.gguf0.99 GBVery aggressive; the last coherent step down.
MiniCPM5-2B-Q2_K.gguf0.97 GBVery small, heavily degraded. For experimentation.
MiniCPM5-2B-Q2KS.gguf0.92 GBSmaller than Q2_K, at a further quality cost.
MiniCPM5-2B-Q2_0.gguf0.87 GBExtreme, group-64. Included for completeness.
MiniCPM5-2B-IQ2_M.gguf0.90 GBThe smallest size most people find usable.
MiniCPM5-2B-IQ2_S.gguf0.85 GBBelow the usual usability line.
MiniCPM5-2B-IQ2_XS.gguf0.80 GBExperimental.
MiniCPM5-2B-IQ2_XXS.gguf0.74 GBExperimental.
MiniCPM5-2B-IQ1_M.gguf0.68 GBExtreme. Expect substantial degradation.
MiniCPM5-2B-IQ1_S.gguf0.64 GBExtreme. Expect substantial degradation.
MiniCPM5-2B-Q1_0.gguf0.50 GBExtreme. Included for completeness.

All 30 files combined: 40.15 GB.

Shortcuts: default Q4_K_M (1.45 GB) · lossless Q8_0 (2.50 GB) · smallest usable IQ2_M (0.90 GB) · 3-bit sweet spot IQ3_M (1.14 GB).

Usage

Run straight from the Hub:

bash
llama-cli -hf NANI-Nithin/MiniCPM5-2B-GGUF:Q4_K_M -p "Hello"

Serve an OpenAI-compatible endpoint:

bash
llama-server -hf NANI-Nithin/MiniCPM5-2B-GGUF:Q4_K_M

Or download a file first:

bash
huggingface-cli download NANI-Nithin/MiniCPM5-2B-GGUF MiniCPM5-2B-Q4_K_M.gguf --local-dir .
llama-cli -m MiniCPM5-2B-Q4_K_M.gguf -p "Hello"

These are llama.cpp GGUF files. They are not loaded by vLLM, SGLang or transformers — for those backends use the source repo.

About MiniCPM5-2B

MiniCPM5-2B is the second model in the MiniCPM5 series from OpenBMB, following MiniCPM5-1B. It is a dense 2B Transformer built for local assistants, coding agents, tool-use workflows and reasoning scenarios where a compact footprint matters. Per the source model card:

  • —2,516,756,480 parameters (1,981,982,720 non-embedding), 42 layers, GQA with 16 query heads and 2 KV heads
  • —131,072-token context with native long-context support
  • —English and Chinese
  • —Trained with the UltraData tiered data management recipe (arXiv:2602.09003): base + mid-training on the released web datasets, then 400B tokens of deep-thinking SFT, RL, and On-Policy Distillation (OPD) of 16 RL teacher models into the release checkpoint
  • —OpenBMB's own evaluation reports 2B-class open-source SOTA with an average score of 53.9 in their comparison set, competitive with 4B-class models, with particular strengths in code reasoning, math, long-context understanding, tool use and agentic tasks

Quantization notes

  • —Quantized from the official BF16 release on 2026-09-07
  • —Importance matrix computed on the BF16 weights, so the IQ and sub-Q4 quants benefit from calibration
  • —All 29 quant types of the standard sweep are present; nothing was skipped for size

Citation

If you use this model, please cite the source work:

bibtex
@article{minicpm4,
  title={Minicpm4: Ultra-efficient llms on end devices},
  author={MiniCPM, Team},
  journal={arXiv preprint arXiv:2506.07900},
  year={2025}
}

Disclaimer

Per the source model card, outputs are generated text and may be inaccurate, biased, or offensive; responses on sensitive topics are not professional advice. The model is provided AS IS, without warranty of any kind; users must use it only for lawful, compliant and ethical purposes.

License

Apache-2.0, matching the source model.