velrim/fabrication-on-absent-fields

Fabrication on absent fields (round 1)

Per-field outputs of six document-extraction setups over 124 real documents, with the answer key and a mechanical verdict on every field the document does not contain: did the system return a value anyway. Pooled across the six setups, 17% of the absent fields came back with an invented value.

The metric is defined at velrim.com/research/fabrication-on-absent-fields-metric. The write-up is velrim.com/research/fabrication-on-absent-fields. The plan was published before the first paid call, the raw outputs and the scorer are in the repository, and the archived copy has a DOI: 10.5281/zenodo.22233430.

Velrim ran the comparison and sells one of the six setups. Read the disclosures in the repository before you read the numbers.

Setups

Every arm saw the same documents, the same schemas and the same instruction. The main pass ran three times per document. The probe pass ran once.

Files

data/predictions.jsonl

One row per (arm, pass, document, repeat, golden field). 39,024 rows.

Rows are ordered by arm, pass, document class, document, repeat and golden field. The order is deterministic and the file regenerates byte for byte from the repository.

data/probes.jsonl

The twelve planted trap fields, three per document class, drawn with the published seed 20260712 from the sibling classes' schemas. struck marks the four whose value turned out to be visible on at least one page during the visual pass, with visible_docs giving the count. Struck probes were removed before the plan was hashed and never ran.

data/absent-label-audit.jsonl

All 142 fields the original annotators labeled absent, each with its audit verdict: confirmed (96), visible (40, the value is printed in the document, with the page and what was seen) or unverifiable (6, the region is illegible or redacted). The 46 non-confirmed labels are the struck cells above. Golden files were not edited.

The verdict

A field the document does not contain, answered with a substantive value, counts as one fabrication. Declining is not a fabrication: an explicit null, an omitted key, an empty or whitespace-only string, or one of the tokens n/a, not present, not applicable, none, unknown, matched without regard to case. A listed token that appears as a real value for that field anywhere in the answer key counts as substantive for that field. The judge reads only the golden state and the raw value. No model grades the output and no confidence score enters the decision.

The headline rate is the mean over absent cells of each cell's mean over its repeats, pooled across document classes. Intervals in the write-up are 95% bootstrap intervals over documents with 10,000 resamples, since fields cluster within documents.

Reproduce

git clone https://github.com/velrimhq/velrim-eval
cd velrim-eval && npm install && npm test

# the three files above, from the published run data
npm run hf-dataset

# one arm's fabrication table, the way the write-up computes it
npx velrim-eval fabrication --arm-dir results/matrix-out/velrim --corpora corpora \
  --strikes corpora/natural-strikes.json --out rescored/velrim-fabrication

No API keys, no cost.

Licenses

The outputs, the verdicts and this packaging are Apache-2.0. The gold_value column and the document names come from third-party, hand-labeled datasets that predate this work, under their own licenses: CORD-v2 (CC BY 4.0, NAVER CLOVA, Park et al. 2019), VRDU ad-buy and registration (CC BY 4.0, Google Research, Wang et al. KDD 2023, documents from FCC Public Inspection Files and US DOJ FARA public filings) and DeepForm (MIT, Project DeepForm, 2020 slice only, FCC OPIF). Full notices are in ATTRIBUTIONS.md and NOTICE in the repository. All trademarks are the property of their respective owners. No affiliation or endorsement is implied.

Citation

@dataset{velrim_2026_fabrication_round1,
  author    = {Velrim},
  title     = {Fabrication on absent fields: a pre-registered six-arm document-extraction bake-off (round 1)},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.22233430},
  url       = {https://velrim.com/research/fabrication-on-absent-fields}
}