useful-quants/LFM2.5-VL-3B-NVFP4-FP8-MinVRAM

LFM2.5-VL-3B (NVFP4 / FP8 Extreme Low-Memory Quantization)

This repository provides the recommended default quantized release of LiquidAI/LFM2.5-VL-3B (pinned upstream commit 5a414ead75d45db003906d06fb62bd5b6846cec0), engineered for NVIDIA GeForce RTX 50-series (Blackwell SM120) high-efficiency serving: it is simultaneously the fastest (leads 5/6 offline workloads) and the smallest of the NVFP4 releases.

[!TIP] Which variant? This repo (MinVRAM) is the default recommendation for general serving: highest throughput (2.21× BF16 decode, 2.55× prefill-heavy, 2.27× C4 aggregate) and the smallest weight footprint (2.60 GiB). Choose useful-quants/LFM2.5-VL-3B-NVFP4-FP8-Mixed only if you need the last word in single-image VQA/OCR response latency (Mixed leads VQA 126.71 vs 117.96 tok/s and OCR 130.66 vs 123.45 tok/s). Do NOT run either variant with --enforce-eager (debug only — it halves throughput; see postmortem in Section 3).

Key Highlights

• —Fastest + Smallest: 163 tok/s decode / 574 tok/s C4 aggregate (2.2× BF16) while compressing weights to 2.60 GiB (-3.22 GiB) — CUDA graph execution required (container default).
• —Aggressive Compression: Extends NVFP4 (W4A4 Block-Scaled E2M1) across all 30 decoder MLPs and all 22 Short-Convolution linear projections; applies FP8 (W8A8 Static E4M3) to attention and vision MLPs.
• —57.1% Payload Reduction: Shrinks the raw tensor checkpoint from 5.82 GiB to 2.49 GiB (2.3332× compression ratio).
• —Maximum VRAM Savings: Reduces model weight footprint to 2.60 GiB, maximizing KV cache allocation on 16GB GPUs.
• —Deterministic Accuracy Parity: Retains exact deterministic accuracy parity on standard VQA, invoice/document OCR, and high-resolution multi-tile visual reasoning benchmarks.
• —Validated Containerized Serving: Ships with a reproducible Docker container based on vLLM 0.26.0 and FlashInfer 0.6.14.

1. Quantization Layout & Precision Policy

LFM2.5-VL-3B utilizes a hybrid linear-attention decoder architecture containing 30 decoder layers:

• —8 Full-Attention Layers (layers.2, 5, 9, 13, 17, 21, 24, 27)
• —22 Short-Convolution Layers (layers.0, 1, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 16, 18, 19, 20, 22, 23, 25, 26, 28, 29)

2. Checkpoint & Storage Accounting

Direct storage manifest audited on host storage:

3. SM120 Hardware Performance Benchmark (CUDA Graph Execution)

Evaluated on NVIDIA GeForce RTX 5060 Ti (16GB GDDR7, SM120) using validated vLLM 0.26.0 with CUDA graph execution enabled (production serving mode; medians of 5 repetitions after warm-up, results/gate7c_sm120_results.json):

Kernel-dispatch profiling (Gate 7C evidence) confirms all NVFP4 linears — including the 22 layers of short-convolution projections unique to this variant — execute on the native CUTLASS FP4 GemmUniversal path with CUDA graph replay. Quantizing those projections removes the BF16 GEMV stream that otherwise dominates decode GPU time, which is why this variant overtakes the Mixed release once graph execution amortizes launch overhead.

[!IMPORTANT] Serving configuration requirement: do NOT use `--enforce-eager`. Postmortem (2026-08): the originally published benchmark table was produced under --enforce-eager and showed this checkpoint at ~0.35× BF16 throughput (21.52 vs 60.48 tok/s decode). That was a serving-configuration defect, not a quantization defect: the NVFP4 decode path issues ~800 small kernel launches per token and becomes CPU-launch-bound under eager execution (GPU ~21% busy) even though its GPU kernel time is ~1.9× faster than BF16. With CUDA graph execution — the container default since 2026-08-18 — this checkpoint reaches 2.21× BF16 decode, 2.55× prefill-heavy, 2.27× C4 aggregate (its weakest relative result is single-image VQA at 1.85× — still well ahead of BF16). ENFORCE_EAGER=1 remains a debug-only escape hatch and prints an explicit performance warning.

4. Serving Environment & Integration Patches

Canonical Recommendation

Use the provided Docker runtime unless you independently apply and validate the compatibility patch suite in your own vLLM build.

The supplied Docker environment pins the validated serving stack and applies four targeted compatibility patches required for this hybrid VLM representation:

1. 1.`patches/0001-flashinfer-backend-binding.patch`
2. 2.Target File: vllm/utils/flashinfer.py
3. 3.Scope: Keyword argument compatibility workaround passing backend="auto" explicitly in flashinfer_autotune_and_run to prevent custom-op parameter collisions on PyTorch 2.11 / SM120.
4. 4.`patches/0002-modelopt-shortconv-resolution.patch`
5. 5.Target File: vllm/model_executor/layers/quantization/modelopt.py
6. 6.Scope: Resolves .conv. $\leftrightarrow$ .short_conv. module routing, handles leaf parameter tensor transposition during safetensors loading, and safeguards missing NVFP4 scale default tensors.
7. 7.`patches/0003-scalar-scale-loader.patch`
8. 8.Target Files:
9. 9.vllm/model_executor/parameter.py
10. 10.vllm/model_executor/layers/linear.py
11. 11.Scope: Broadcasts 0D scalar scale tensors to multidimensional parameter containers and safeguards weight parameter container loading.
12. 12.`patches/0004-shortconv-rank-alignment.patch`
13. 13.Target Files:
14. 14.vllm/model_executor/layers/mamba/short_conv.py
15. 15.vllm/model_executor/models/lfm2.py
16. 16.vllm/model_executor/models/lfm2_siglip2.py
17. 17.vllm/model_executor/models/lfm2_vl.py
18. 18.Scope: Passes quant_config through ShortConv instantiation, aligns 1D depthwise convolution tensor shapes, and adds WeightsMapper prefix translations (vision_model.) for SigLIP2 vision encoder weights.

Zero Modification Statement: These compatibility adaptations do not recalibrate the checkpoint, alter learned tensor values, change model topology, or modify the mathematical inference equations. All 86 FP8 scale tensors match the frozen PTQ calibration bit-for-bit (0 mismatches, max absolute difference = 0.0).

Version Support Policy

• —Validated Runtime: vLLM 0.26.0 + PyTorch 2.11.0+cu130 + FlashInfer 0.6.14 (as packaged in the included Dockerfile).
• —Other Versions (e.g. vLLM 0.27.x): UNVALIDATED. Newer versions have not been qualified against this specific checkpoint representation and SM120 configuration.

5. Quick Start with Docker

Step 1: Clone Repository & Build Serving Image

git clone https://huggingface.co/useful-quants/LFM2.5-VL-3B-NVFP4-FP8-MinVRAM
cd LFM2.5-VL-3B-NVFP4-FP8-MinVRAM
docker build -t lfm25vl-serving:v0.26.1 .

Step 2: Run Serving Container

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v "$(pwd):/model" \
  -v lfm-jit-cache:/root/.cache \
  -e MODEL_PATH=/model \
  lfm25vl-serving:v0.26.1

# CUDA graphs are ON by default (required for rated throughput).
# The container performs a startup warm-up (text + image) after /health passes,
# absorbing per-shape Triton JIT before client traffic. WARNING: the FIRST boot
# of a given model/spec compiles FlashInfer CUTLASS kernels into /root/.cache -
# measured ~75 minutes on the validation host (12-core CPU) - and the server
# does not become healthy until it finishes. Mount a persistent volume (as
# above) so this one-time cost is paid once per model/spec, not per container.
# ENFORCE_EAGER=1 disables graphs (debug only, ~2-5x slower).

Step 3: Run Deterministic Smoke Suite

# Text Generation Smoke Test
python examples/smoke_text.py --url http://localhost:8000/v1/chat/completions

# Multimodal VQA Smoke Test
python examples/smoke_image.py --url http://localhost:8000/v1/chat/completions

6. Notice of Modifications & License Compliance

• —Base Model: LiquidAI/LFM2.5-VL-3B (pinned upstream revision 5a414ead75d45db003906d06fb62bd5b6846cec0) by Liquid AI, Inc.
• —License: LFM Open License v1.0 (Liquid AI, Inc.).
• —Modifications Notice: See MODIFICATIONS.md for an itemized record of post-training quantization, runtime patches, and derivative work notices in accordance with Section 4 of the LFM Open License v1.0.
• —Attribution: Quantization, patch derivation, containerization, and release packaging created by useful-quants. Liquid AI, Inc. did not produce, sponsor, or endorse this derivative work.