Modelpublic

OsaurusAI/LFM2.5-2.6B-MXFP8

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

0likes95downloads

Model Card

OsaurusAI/LFM2.5-2.6B-MXFP8

MXFP8 build of LiquidAI/LFM2.5-2.6B — LiquidAI's always-thinking 2.6B agentic model (16 languages, 128K context) in the OCP microscaling FP8 format, with learned codes: activation-aware AWQ folds plus GPTQ codes-only QAT on every FFN tensor. Same size as the vendor's own MXFP8 export, lower KL in both measured domains.

Want the best quality per GB? Take `LFM2.5-2.6B-JANG_6M` (2.30 GiB, KL 0.033/0.0033 bits vs this build's 0.272/0.113). Choose MXFP8 when you want the OCP microscaling format end-to-end.

Bundle

Field	Value
Source	`LiquidAI/LFM2.5-2.6B` @ `dca1825` (LFM 1.0 license)
Architecture	`lfm2` dense hybrid — 22 double-gated short-conv (LIV) blocks + 8 GQA attention layers, 2.69B params, 128K ctx
On-disk size	2.59 GiB (1 shard)
Quantization	every 2-D weight MXFP8 (`mx.quantize mode="mxfp8"`): e4m3 codes + e8m0 scales, group size 32
AWQ	per-channel folds (α 0.25, clip 0.5–2.0) into `ffn_norm`→w1/w3 and w3-rows→w2 — function-preserving, zero runtime cost
QAT	GPTQ codes-only learned rounding of the e4m3 codes on the fixed e8m0 scale grid, all 90 FFN tensors, BRECQ-sequenced w1/w3→w2, best-of-RTN guard, byte-parity with `mx.quantize` verified at build — mean recon error −69.7% vs RTN (`qat_report.json`)
Calibration	canonical mix rendered through the chat template with greedy thinking continuations (10.4K tokens/layer)
Norms, conv kernels	fp16 passthrough — plain Llama RMSNorm, no +1 shift
Attention	32 heads / 8 KV heads (GQA), head_dim 64, per-head q/k RMSNorm, NeoX RoPE θ = 1e7
Modality	text-only (verified from the tensor index — no vision/audio weights; the template's `<image>` item handling is inert on this model)

Measured (M5 Max, stock mlx-lm 0.31, vs bf16 source)

Two 768-token held-out texts: general/encyclopedic and agentic/code+thinking. KL is mean full-vocabulary KL(bf16 ‖ quant).

Bundle	Size	Top-1 (gen / agentic)	Mean KL bits (gen / agentic)	Decode
`LFM2.5-2.6B-MXFP8` (this)	2.59 GiB	91.1% / 93.4%	0.272 / 0.113	146 tok/s
vendor MLX mxfp8 (RTN)	2.59 GiB	91.7% / 92.7%	0.303 / 0.115	—
`LFM2.5-2.6B-JANG_6M`	2.30 GiB	97.3% / 98.4%	0.033 / 0.0033	156 tok/s
bf16 source	5.02 GiB	100%	0	83 tok/s

As on every model we have measured, 6-bit affine beats MXFP8 on fidelity while being smaller — e4m3 elements carry ~3 mantissa bits, so "8-bit MX" is not strictly better than 6-bit affine with a per-group scale and bias. It shows in behaviour too: under pure greedy decoding this build thinks more verbosely than bf16 (it re-verifies its own arithmetic before answering — still correct, </think> closes, eos fires); with the card's default sampling (temperature 0.1) outputs are clean and concise. Runtime gates all pass: greedy math reasoning, card-default sampling coherence, Liquid-format tool calls, grounded 2K-token long-context answers.

Chat / reasoning

—Thinking is ALWAYS on. LFM2.5-2.6B is a pure reasoning model: the chat template unconditionally opens <think> at the start of every assistant turn. There is no enable_thinking switch — the only template kwarg is preserve_thinking (default false: prior turns' reasoning is stripped except after the last user turn).
—The template is shipped verbatim (chat_template.jinja, also inlined into tokenizer_config.json), and capabilities.think_in_template = true is stamped so think-tag parsers route the pre-opened block correctly.
—No BOS trap: the template emits <|startoftext|> itself and the tokenizer never auto-adds one — both apply_chat_template(tokenize=True) and re-encoding the rendered string yield exactly one BOS.
—Stop token eos_token_id = 124900 (<|im_end|>).
—Tool calls use the Liquid Python-call format: <|tool_call_start|>[get_weather(city='Seoul')]<|tool_call_end|> (verified live).
—Sampling defaults (vendor card + generation_config.json, mirrored in jang_config.chat.sampling_defaults and gate-checked against each other at build): temperature 0.1 · top_k 50 · repetition_penalty 1.1.

Usage

Standard MLX safetensors with {"group_size": 32, "bits": 8, "mode": "mxfp8"} in config.json[quantization]. Loads with stock `mlx_lm >= 0.31` — no custom code, no trust_remote_code. Runs in Osaurus and vMLX-compatible runtimes (lfm2 family).

python

from mlx_lm import load, generate

model, tokenizer = load("OsaurusAI/LFM2.5-2.6B-MXFP8")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Which number is bigger, 9.11 or 9.8?"}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=1024))