Datasetpublic

shubhambaid/lfm2vl-lab-results

lfm2vl-lab — raw evaluation results Every number behind lfm2vl-lab, a component-wise quantization and edge-inference study of LiquidAI/LFM2.5-VL-3B on Apple Silicon (M4 Pro, 24 GB). Published separately from the code so the numbers stay citable and diffable on their own. Weights in that study are ephemeral by design — converted, evaluated, recorded, deleted. These JSON files are what survives. Files file what it holds coverage_audit_gs64.json per-module… See the full description on the dataset page: https://huggingface.co/datasets/shubhambaid/lfm2vl-lab-results.

sourceHugging Facemitupdated 1mo agoView on Hugging Face

0likes28downloads

Dataset Card

lfm2vl-lab — raw evaluation results

Every number behind `lfm2vl-lab`, a component-wise quantization and edge-inference study of `LiquidAI/LFM2.5-VL-3B` on Apple Silicon (M4 Pro, 24 GB).

Published separately from the code so the numbers stay citable and diffable on their own. Weights in that study are ephemeral by design — converted, evaluated, recorded, deleted. These JSON files are what survives.

Files

file	what it holds
`coverage_audit_gs64.json`	per-module quantizability audit at group size 64
`coverage_audit_by_group_size.json`	the same at group size 32 / 64 / 128
`uniform4_coverage_verified.json`	coverage read back from saved weights
`published_quant_survey.json`	56 published LFM2-VL/LFM2.5-VL quant repos, checked for vision-tower quantization
`token_budget_sweep.json`	accuracy and latency vs `max_image_tokens`, 3 tasks × 4 budgets
`baseline_bf16_n200_t512.json`	bf16 baseline vs Liquid's published figures
`sweep_component_sensitivity.json`	17 quantization configs × 5 benchmarks
`head_to_head_n300.json`	5 decision-relevant configs at n=300 with 95% CIs

Headline findings

The vision token budget matters more than quantization. ScreenSpot-v2 moves from 53.0 to 79.0 by changing max_image_tokens from the default 256 to 512 — and falls back to 70.0 at 1024. Non-monotonic, and the optimum is task-dependent (512 for screens and documents, 256 for natural images).

Component sensitivity is flat. Across 17 configs, nothing measurably degrades at 6 or 8 bits on any component. The only clear loser is uniform 4-bit, the one config pushing the vision tower to 4 bits — lower on 5 of 5 benchmarks.

A third of the vision tower cannot be quantized. SigLIP2 shape-optimized has intermediate_size = 4304 (16 × 269, 269 prime), so all 27 mlp.fc2 modules are skipped at every standard group size — silently, with no warning.

No published quant touches the vision tower. Of 56 repos surveyed, including LiquidAI's own MLX releases, none has a vision_tower quantization entry.

Caveats

n is 100–300 per benchmark, so 95% CIs run ±4 to ±5 points. Differences smaller than that are not claims. Single machine, single seed (0), mlx-vlm==0.6.13.