nutrientdocs/document-classification-benchmark

Document Classification Benchmark (open-vocab, zero-shot)

Given a document image and an arbitrary set of text labels, which one is right? A held-out, zero-shot, open-vocabulary evaluation for document-type classification — labels are supplied at inference, not baked into a head. Test split only; not for training. Every image is drawn from a permissively-licensed, redistributable source.

Powers the document-classification-leaderboard and evaluates `document-classification-v2` (commercial) and the open-weight `document-classification-v1`; try it live in the demo.

Tracks (macro-F1)

• —DocLayNet — document page categories (financial reports, scientific articles, manuals, patents, …).
• —Forms — form vs non-form and fine form sub-types.
• —OOD — document types absent from training (invoices, handwriting, charts, tables) — the open-vocab stress test.
• —OOV — an evaluation protocol (not extra rows): score the same images against never-seen synonym paraphrases of each label ("invoice" → "bill") to measure concept-matching vs memorized wording.

Contents & sources

897 test images (track column selects the slice). All sources are redistributable:

Tobacco3482 is deliberately excluded. Its research-only license is not redistributable, so it is not part of this public benchmark (it is used only in internal evaluation).

from datasets import load_dataset
ds = load_dataset("nutrientdocs/document-classification-benchmark", split="test")

Schema

About the author

<a href="https://nutrient.io/"> <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" /> </a>

This project is maintained and funded by Nutrient - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.