nutrientdocs/form-field-vlm-v2-benchmark

form-field-vlm-v2-benchmark

A balanced held-out slice for evaluating form-field detect + extract — detection + fine type + text label + entered value — across the three render conditions a real pipeline meets: empty, filled (printed), and handwritten. Drawn from the commonforms-synth-v2 split (document-disjoint from training).

• —📊 Benchmark: nutrientdocs/form-field-vlm-v2-benchmark
• —🎯 Model: `nutrientdocs/form-field-vlm-v2` · 🏆 Leaderboard

Composition

354 pages — 113 empty / 116 filled / 125 handwritten — type-stratified so radio / checkbox / select / signature are each well represented. 8,220 gold fields; roughly half carry a non-empty entered value (the extraction target on the filled and handwritten pages). Gold per field: bbox (original-page [x,y,w,h]), field_type, label_text, description, value, group_id, state.

Metric

Predictions are matched to gold greedily by IoU ≥ 0.5. The headline is End-to-end F1: a field counts only when the box (IoU ≥ 0.5), the type, and the value are all correct. Reported alongside are the sub-skills — Detect (IoU + type only), Value and Label (over correctly-detected fields). All metrics are macro-averaged over field types. Free-text values and labels are scored by multilingual bge-m3 embedding cosine; choice values (checked / unchecked / option) and the strict value_exact diagnostic use exact match. See scripts/score_extract_ffde.py for the reference scorer.

<!-- RESULTS-TABLE:START --> | System | End-to-end | Detect | Value | Label | |---|---:|---:|---:|---:| | Nutrient Hybrid (detector + VLM) | 0.565 | 0.628 | 0.917 | 0.713 | | Nutrient VLM v2 | 0.518 | 0.550 | 0.952 | 0.775 | | GPT-5.6 Sol (best cloud) | 0.338 | 0.461 | 0.689 | 0.674 | <!-- RESULTS-TABLE:END -->

Full 11-model comparison (nine cloud VLMs + Nutrient VLM + hybrid), per-render-condition breakdowns, and diagnostics on the leaderboard.

Coordinates: gold boxes are original-page [x,y,w,h]; predictions use the documented 0–1000 grid.

Notes

• —Sample the whole slice, not the head (the parent split is stored empty-ordered).
• —description gold is VLM-annotated and is a diagnostic only — not part of the headline ranking.
• —Source: CommonForms-derived pages; license CC-BY-4.0.

About the author

<a href="https://nutrient.io/"> <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" /> </a>

This project is maintained and funded by Nutrient - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.