CoolFace
Modelpublic

rdtand/LFM2.5-8B-A1B-PrismaQuant-6.5bit-vllm

sourceHugging Faceupdated 4mo agoView on Hugging Face
0likes31downloads
Model Card

PrismaQuant LFM2.5-8B-A1B (mixed-precision, ~6.58 bpp)

Post-training mixed-precision quantization of LiquidAI/LFM2.5-8B-A1B, produced with PrismaQuant — a per-Linear format allocator that places each weight matrix at the cheapest format its KL-sensitivity allows, then exports to compressed-tensors.

Summary

Precisioncompressed-tensors / mixed-precision
Average bits/param~6.58 bpp (allocator post-cliff log knee)
On-disk size8.35 GB
Quantized-Linear mixNVFP4 (4b) ×27, FP8_E4M3 (8b) ×20, BF16 ×27
BF16 passthroughshort-conv in/outproj, all RMSNorms, router + expertbias, embed/lm_head, plus a few sensitive expert layers
end-KL vs BF160.131 nats/token (PrismaQuant harness)
ToolEvalBench (74 hard-mode)70/100 (104/148) — identical to the BF16 source
vLLM serving✅ loads + generates coherently (matches BF16)

Evaluation

ToolEvalBench agentic tool-call benchmark, 74 hard-mode scenarios (--temperature 0 --seed 1234 --timeout 180 --parallel 1), served through vLLM with the lfm2 tool-call parser:

This (6.58 bpp NVFP4)BF16 source
Score70 / 10070 / 100
Points104 / 148104 / 148

Quantization is capability-neutral here — the 4-bit MoE experts match the BF16 source on aggregate agentic tool use (and on Tool Selection, Restraint & Refusal, and Error Recovery the quantized model scores 100%). Per-category differences are within single-run greedy noise; the only category where the quant trails is Safety & Boundaries (20/26 vs 24/26).

The MoE experts (≈93% of params) carry the compression and are mostly NVFP4 (4-bit); sensitive paths stay FP8/BF16.

Serving (vLLM)

bash
vllm serve <this-dir> --quantization compressed-tensors --trust-remote-code

Requires a vLLM build with Lfm2MoeForCausalLM and the LFM2 short-conv / linear-attention kernels (causal-conv1d, flash-linear-attention). Use the model's chat template — it's an instruct/reasoning model (raw-text completion will look degenerate; this is the base model's behavior, not a quantization artifact, and is identical to the BF16 source).

Method

  • —Probe — per-Linear diagonal-Fisher KL-sensitivity over calibration text.
  • —Cost — production-render-score: each (Linear, format) is rendered with its real export levers (GPTQ, joint-scale-opt, static-act-order) and scored by output MSE.
  • —Allocator — Pareto DP over per-Linear (format, cost) with a post-cliff log-error Kneedle pick; packed-MoE experts compete per-format.
  • —Calibration — diverse-v1, 8 samples × 1024 tokens.
  • —Built directly from the source per-expert checkpoint — PrismaQuant packs experts into the live module on the fly (no pre-pack).

Notes

  • —Quantization-only; weights are not further fine-tuned.
  • —end-KL (0.131 nats/token vs BF16) is the meaningful quality signal. Absolute raw-text perplexity is high/method-dependent (instruct model on raw text), not a clean quantization metric.
  • —The short-conv mixer Linears are kept BF16 (vLLM builds ShortConv without a quantconfig), and NVFP4 experts are emitted under vLLM's canonical `gateproj/upproj/downproj` scheme names so the FusedMoE scheme detector binds correctly.

Attribution: robert.tand@icloud.com