ljupco/cpu-only-inference-models

CPU-Only Inference Models

Small, efficient LLMs that run well on ordinary CPUs — no GPU needed. All models below were measured on a modest 2019-era laptop CPU:

thinkpad2: Intel Core i5-8350U (4C/8T, AVX2), 64 GB DDR4-2400, AC power, 4 threads.

If your machine has no discrete GPU (or you want to keep the GPU free), these are the models and setups that actually work — with real token/s numbers, not promises.

⭐ The star: Maple Preview 20B-A1B (ternary 2-bit)

The fastest useful model on a CPU we have found — over 28 tokens/s on a 4-core laptop.

Maple Preview is DeepGrove's open-source reasoning model, designed from the start for efficient on-device inference (24 layers, 3:1 SWA-512:GA attention, 131k context, MIT license). It is the real star of this collection: on our CPU it runs at 28–34 tokens/s — comfortably interactive — while its 20B total / 1B-active ternary weights keep it to a 5.5 GB file that fits any machine with 16 GB of RAM.

Reference points (from the DeepGrove team and our own measurements):

• —this CPU (i5-8350U, 4 threads): prompt 512 tok → 100.8 t/s · decode tg128 → 33.9 t/s
• —Apple M2 Max CPU: ~360 t/s prompt · ~77 t/s decode
• —Apple M4 mini (DeepGrove's measurement): 200+ t/s

The engine is the DeepGrove llama.cpp fork (the maple architecture + TQ20 support); the GGUF we use is their `maple-preview-TQ20-head-Q4_K.gguf`.

• —Model: deepgrove/maple-preview (MIT)
• —GGUF: deepgrove/maple-preview-GGUF
• —Engine: deepgrove-ai/llama.cpp
• —Announcement: deepgrove on X

llama-server -m maple-preview-TQ2_0-head-Q4_K.gguf \
  --ctx-size 131072 --cache-type-k q8_0 --cache-type-v q8_0 \
  --threads 4   # 4 beats 8 on this CPU (33.9 vs 21.8 t/s)

LiquidAI LFM2.5 family — mixed 4-bit GGUFs by ljupco

The LFM2.5 models (2.6B dense, 1.2B-Thinking, 8B-A1B MoE) with mixed quantizations: the bulk of the weights at 4-bit, the most sensitive tensors kept at higher precision.

Original models by Liquid AI — LFM2.5-2.6B, LFM2.5-1.2B-Thinking, LFM2.5-8B-A1B.

llama-server -m LFM2.5-2.6B-Q4_K_M.gguf --threads 4

The full three-engine report

These numbers come from a systematic porting and benchmarking exploration across three engines — stock llama.cpp (with the DeepGrove fork for Maple), ik-llama.cpp, and vllm.cpp — including kernel-level work (fused ops, integer-dot kernels, a ternary gemv) and a detailed analysis of why the DeepGrove 8x8 gemv cannot run on standard-quantized weights:

[The LFM2.5 / Maple Preview three-engine report](docs/lfm2-maple-4bit-report.md)

Credits and Acknowledgements

This collection is entirely built on the work of others, and we are deeply grateful:

• —DeepGrove AI — for the Maple Preview model, the TQ2_0 ternary quantization, and their llama.cpp fork with the Maple architecture support. The ternary design and the on-device focus are what make 28+ t/s on a CPU possible. Thank you!
• —Liquid AI — for the LFM2.5 family and its gated-delta / shortconv architecture, and for publishing the weights openly. Thank you!
• —llama.cpp / ggml — the core inference engine and its maintainers and contributors.
• —The HuggingFace / GGUF ecosystem for the format, the tooling, and the platform.

Any remaining errors are ours. Benchmark numbers are single-machine measurements; expect ±10–20% day-to-day noise.