lilcheaty/LFM2.5-230M-MLX-attn8-base6

LFM2.5-230M-MLX-attn8-base6

A custom mixed-precision MLX quant of Liquid AI's LFM2.5-230M: 6-bit base weights + 8-bit attention. On an M4 MacBook Air it runs ~17% faster than stock 8-bit at identical quality — the speed of a 6-bit model with the correctness of 8-bit.

A faster sibling, `attn8_base5` (5-bit base + 8-bit attention, ~35% faster than 8-bit, same quality), lives on the `attn8_base5` branch.

Why mixed precision

Quantizing LFM2.5 uniformly trades quality for speed, and the damage is concentrated in specific weights. Per-task probing on extraction / classification / factual prompts showed:

• —Attention (q/k/v/out, ~5% of params) must stay 8-bit. This is what controls "extract directly" vs "write a regex script." Drop it to 6-bit and the model starts dumping Python regex instead of answering an "extract the emails" instruction.
• —Embeddings (the tied lm_head / logit matrix, ~29% of params) need ≥5-bit. At 4-bit they corrupt both extraction and factual recall.
• —Everything else (the ~48% MLP, conv blocks, embeddings) tolerates 6-bit fine.

So this model keeps attention at 8-bit and everything else at 6-bit (group size 64, affine). It's effectively "the known-good 6-bit model, patched at exactly the one capability 6-bit broke" — the smallest change that recovers full 8-bit quality while beating 8-bit speed. (config.json records the per-layer bits: 24 layers @ 8-bit, 59 @ 6-bit.)

Benchmarks (M4 MacBook Air 16 GB, mlx-lm 0.31.3, greedy, 256 tok)

Speed = median of 3 timed runs. Quality = a fixed set of 6 checkable extraction/classification/factual tasks; an arithmetic task all variants fail is excluded — this isn't a math model.

Before → after: stock 8-bit @ ~325 tok/s → this model @ ~381 tok/s (+17%) at the same 6/6 quality. Numbers are machine-specific; reproduce with the recipe below.

Usage

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

# default branch = attn8_base6 (6-bit base + 8-bit attention)
model, tok = load("lilcheaty/LFM2.5-230M-MLX-attn8-base6")

# faster sibling (5-bit base + 8-bit attention):
# model, tok = load("lilcheaty/LFM2.5-230M-MLX-attn8-base6", revision="attn8_base5")

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Extract all emails from: 'Reach ana@acme.io or sales@acme.io'. Comma-separated."}],
    add_generation_prompt=True,
)
print(generate(model, tok, prompt, max_tokens=128, sampler=make_sampler(temp=0.0)))

CLI:

python -m mlx_lm generate --model lilcheaty/LFM2.5-230M-MLX-attn8-base6 \
  --prompt "Summarize the benefits of on-device LLMs." --max-tokens 128

How it was built

from mlx_lm.convert import convert

def predicate(path, module):
    # attention at 8-bit, everything else at 6-bit; group size 64
    bits = 8 if "self_attn" in path else 6
    return {"group_size": 64, "bits": bits, "mode": "affine"}

convert(
    hf_path="LiquidAI/LFM2.5-230M-MLX-bf16",
    mlx_path="attn8_base6",
    quantize=True, q_group_size=64, q_bits=6,
    quant_predicate=predicate,
)

Intended use & limitations

LFM2.5-230M is a small extraction / classification / lightweight-agentic model — great for data extraction, structured output, and on-device tasks. It is not for heavy reasoning, math, or long-form creative writing (it cannot reliably do multi-digit arithmetic, by design). This quant preserves the base model's capabilities and limits.

License & attribution

Derivative of [LiquidAI/LFM2.5-230M](https://huggingface.co/LiquidAI/LFM2.5-230M) (© Liquid AI), redistributed under the LFM Open License v1.0 (`lfm1.0`). All credit for the model goes to Liquid AI; this repo only re-quantizes the weights for MLX. Please review the base model's license for your use case.