Mitchins/acme-home-inbox

ACME Home Inbox

Public dataset: <https://huggingface.co/datasets/Mitchins/acme-home-inbox>

The v0.2.0 checkpoint contains the complete synthetic dataset plus the first canonical OCR/vision deployment bake-off. Benchmark results are an auditable research checkpoint, not a claim that any tested routing policy is ready for unattended household use.

ACME Home Inbox is a fully synthetic, reproducible stress test for a practical systems question:

Does graphical evidence change the correct household decision, rather than merely produce a richer description?

It contains 64 fictional Australian household-inbox cases in four tiers. Cases cover bills, receipts, government and school correspondence, health documents, handwritten notes, photographs, mixed scans, and safe-abstention controls. The fictional household is fixed so ownership decisions are deterministic.

This is an agentic document-routing benchmark, not a general-purpose VLM leaderboard. It is designed to compare OCR-only, OCR-plus-layout, OCR-plus-visual-sidecar, image-only, and image-plus-OCR decision paths.

Dataset contents

Each case includes:

• —one or more scan-like page images;
• —its deterministic source PDF;
• —unedited Tesseract plain text, TSV word boxes, and hOCR;
• —a household decision in the published JSON schema;
• —bounded visual-sidecar evidence labels;
• —the facts rendered into the source document;
• —image, PDF, OCR, prompt-independent provenance and hashes;
• —tier, visual-target, degradation, and lockbox metadata.

Files use opaque identifiers such as acme_0033_p01.jpg. Categories and answers do not appear in filenames.

The image folders are compatible with Hugging Face imagefolder: each split contains a metadata.jsonl beside its page images. Pages belonging to the same case share case_id and are ordered by page_index. Scalar gold fields are direct columns; nested structures use explicit *_json strings so feature types remain identical between splits.

from datasets import load_dataset

dataset = load_dataset("imagefolder", data_dir="data/images")

The canonical case-level manifest is data/manifests/cases.jsonl. data/manifests/checksums.sha256 covers every published dataset artefact.

Fixed fictional household

• —Alex Morgan — adult
• —Jamie Morgan — adult
• —Riley Morgan — child
• —Milo — dog
• —Home — 14 Acacia Street, Greenfield NSW 2000
• —Vehicle — Toyota Corolla, registration 1ABC23
• —Business — Acacia Consulting Pty Ltd
• —Household — shared family material
• —Unknown — insufficient evidence; abstention required

All names, issuers, ABNs, account numbers, addresses, claim numbers, events, amounts, and documents are fictional. No private source document was used.

Tiers

The evaluation date is fixed at 2026-08-03. Urgency labels must be judged relative to that date rather than wall-clock time.

Splits

The logical 48-case dev split is intended for harness and prompt development; it is published as the Hugging Face validation split. The logical 16-case lockbox is published as test. It was selected before model evaluation and contains four cases from each tier:

04, 08, 13, 16, 20, 24, 29, 32, 36, 40, 44, 48, 58, 59, 62, 63

The Tier-4 lockbox deliberately covers front/back linking (58), safe abstention (59), an immediate safety case (62), and multi-page splitting (63), rather than relying on tier balance alone.

Do not use any of the 64 cases for quantisation calibration, QAT, adapter training, OCR correction rules, or prompt fitting.

Intended benchmark arms

The visual sidecar is evidence-only. It does not decide household policy.

Reference bake-off

The repository includes a pinned, bounded reference experiment using official GPT-OSS-20B MXFP4, Florence-2-large-ft FP16, and Gemma 4 E4B QAT Q4_0 GGUF weights. Its principal result is that native images help the difficult visual tail but are not proven superior as an unconditional input. The highest observed policy uses Gemma OCR-first and invokes its native image path for multi-page or sparse-OCR scans; it is explicitly exploratory because its gate was selected on development data and its lockbox uplift was inconclusive.

The complete tally, uncertainty intervals, speed measurements, backend matrix, sidecar task-level correction, and deployment recommendation are in `results/reports/bakeoff-findings.md`. Exact isolated-environment and execution commands are in `docs/bakeoff-reproduction.md`. Raw model responses and every length replay are retained in results/bakeoff/; model payloads are excluded.

Florence-2 is evaluated through its native task prompts. A <MORE_DETAILED_CAPTION> result wrapped as uncertain evidence must not be described as though the model followed prompts/visual-sidecar-system.txt. Always-on captions, heuristic invocation, and semantic uncertainty/review invocation are reported separately.

Outcome schema and scoring

The exact decision and visual-sidecar schemas are in schemas/. Scoring is weighted toward household consequences:

• —relation/type: 25%;
• —owner: 20%;
• —urgency/action: 25%;
• —filing and secondary routes: 25%;
• —split/group/abstention: 5%.

The implementation adds document_structure.items to the brief's base schema so each document in a mixed scan has its own bound relation, owner, urgency, route, and action. This prevents an aggregate answer from receiving credit for recognising two documents without routing both items correctly. Canonical relation and filing-route labels are published in config/label-taxonomy.json and should be included with the decision prompt.

Safety and unsupported-claim penalties are additional. In particular, under-calling U4 is a case-level safety failure, guessing owners for ambiguous cases is an abstention failure, and changing a correct Tier-1 decision is a vision-harm event.

Official aggregate scores require complete case coverage. The scorer separately reports schema failures, unresolved truncations, heavy repetition, vacuousness, token diversity/entropy, runaway reasoning, and operational length loops; fenced JSON and supported thinking wrappers remain valid when the enclosed object conforms to schema.

Generation

The renderer creates distinct vector source documents with realistic Australian terminology, dates, GST presentation, fictional business details, tables, stamps, checkboxes, barcodes, signatures, annotations, and page structure. It then rasterises those PDFs and applies deterministic scan conditions. OCR is run against the final raster, never against source text.

Lucide icons and OFL-licensed fonts are pinned and hashed. Five photographic inserts were generated specifically for the suite; exact benchmark text is overlaid deterministically rather than delegated to an image generator. See docs/ATTRIBUTION.md and assets/generated/manifest.json.

Rebuild from a clean checkout:

make setup
make all

Tesseract and Poppler executables must be available. Exact tool versions used for the published build are recorded in data/manifests/build.json.

Limitations

• —The suite is synthetic and tests workflow decisions, not population-level document prevalence.
• —OCR difficulty is renderer- and Tesseract-version-specific.
• —Handwriting is a mixture of bundled handwriting fonts and procedural marks, not a demographic sample of human handwriting.
• —A correct answer does not imply legal, financial, medical, or safety advice.
• —The fictional identifiers are visually realistic but must never be used as real contact, payment, account, or government identifiers.

Licences

Dataset artefacts and labels are released under CC BY 4.0. Generation and validation code is released under Apache 2.0. Third-party asset licences are preserved in assets/ and documented in docs/ATTRIBUTION.md.