Datasetpublic

datalab-to/omni_extract_bench

Omni Extract Bench We weren’t satisfied with the current benchmarking options for extraction. They were biased, didn’t use realistic data and were hard to audit. Our view is that an extraction benchmark should do two things: Help customers choose the right vendor; and Give engineers a way to diagnose what’s actually going wrong in a given model. That’s why we built OmniExtractBench. OmniExtractBench is a comprehensive structured extraction benchmark, developed by Datalab.… See the full description on the dataset page: https://huggingface.co/datasets/datalab-to/omni_extract_bench.

sourceHugging Facecc-by-4.0updated 3d agoView on Hugging Face

3likes1.6kdownloads

Dataset Card

Omni Extract Bench

We weren’t satisfied with the current benchmarking options for extraction. They were biased, didn’t use realistic data and were hard to audit. Our view is that an extraction benchmark should do two things:

—Help customers choose the right vendor; and
—Give engineers a way to diagnose what’s actually going wrong in a given model.

That’s why we built OmniExtractBench.

OmniExtractBench is a comprehensive structured extraction benchmark, developed by Datalab. It tests how accurately models pull specific values out of documents when given a schema and a PDF. OmniExtractBench includes:

—620 diverse documents from several vendor benchmarks (Reducto, Extend, LlamaIndex, Datalab) to eliminate bias;
—Clear, consistent and auditable scoring; and
—A wide variety of extraction edge cases tested (scans, dense tables, forms, etc)

Read more in our blog post. Also see the GitHub for the scoring and prediction harnesses.

Data

[image]

Results

[image]

Layout

manifest.parquet        one row per document; the schema is in it
pdfs/<doc_id>.pdf       the document
gold/<doc_id>.json      the ground-truth extraction
licenses/               the upstream licence for each suite
assets/

column
`doc_id`	unique; a descriptive name, which the files carry. Some contain spaces
`doc_path`	`pdfs/<doc_id>.pdf`, relative to the dataset root
`gt_path`	`gold/<doc_id>.json`, relative to the dataset root
`schema`	the JSON Schema itself, inline as bytes
`suite`	which part of the benchmark it came from

suite names the part of the benchmark a document came from, and rides through to your scores, so results can be read per suite as well as overall.

Install

bash

uv pip install omni-extract-bench                 # the scorer: score(), scipy and nothing else
uv pip install 'omni-extract-bench[harness]'      # + vendor adapters, to produce predictions
uv pip install 'omni-extract-bench[benchmark]'    # + packages to orchestrate and run benchmark

See our Github to see the code and for more details.

[image]

Run our benchmark with one line

We provide orchestration to run our benchmark around our core primitives: predict and score. Run the benchmark (limit to 1 document here). It's resumable so you can stop and reinvoke to resume at any point.

!!NOTE!!: this will cost money and you will need your API keys set.

bash

oeb benchmark --out runs/ --limit 1 \
    --providers datalab reducto extend llamaextract

You can also specify settings per provider. For example:

bash

oeb benchmark \
  --providers datalab reducto \
  --limit 1 \
  --options '{"datalab":  [{"mode": "balanced"}, {"mode": "accurate"}],
              "reducto": [{"agentic_table_mode": "max"},
                          {"agentic_table_mode": "default"}]}' \
  --out runs/

This will execute 4 different runs -- one for each pair (provider, settings).

See `docs/API.md` for more details on what oeb benchmark writes and how you can run on your own benchmark dataset.

Score

Use in your own code.

python

from omni_extract_bench import score

schema = {
  "type": "object",
  "properties": {
    "invoice_id": {
      "type": "string",
      "description": "The invoice number as printed on the document."
    },
    "invoice_date": {
      "type": "string",
      "description": "Date of issue, ISO 8601 (YYYY-MM-DD)."
    },
    "total_due": {
      "type": "number",
      "description": "Total amount payable, in the invoice currency."
    },
    "purchase_order": {
      "type": "string",
      "description": "Buyer's purchase order number, where the invoice cites one."
    },
    "line_items": {
      "type": "array",
      "description": "One entry per billed line.",
      "items": {
        "type": "object",
        "properties": {
          "sku": {"type": "string", "description": "Stock code for the item."},
          "description": {"type": "string", "description": "Item description as printed."},
          "qty": {"type": "integer", "description": "Units billed."},
          "unit_price": {"type": "number", "description": "Price per unit."}
        }
      }
    }
  }
}

prediction = {
  "invoice_id": "INV-4417",
  "invoice_date": "03/31/2024",
  "total_due": "1,240.00",
  "purchase_order": "PO-88231",
  "currency": "USD",
  "line_items": [
    {"sku": "BX-2201", "description": "Washer, 8mm", "qty": 100, "unit_price": 9.4},
    {"sku": "AX-9910", "qty": 24, "unit_price": 12.05},
    {"sku": "ZZ-0000", "description": "Freight surcharge", "qty": 1, "unit_price": 45.0}
  ]
}

ground_truth = {
  "invoice_id": "INV-4417",
  "invoice_date": "2024-03-31",
  "total_due": 1240.0,
  "line_items": [
    {"sku": "AX-9910", "description": "Hex bolt, M8", "qty": 24, "unit_price": 12.5},
    {"sku": "BX-2201", "description": "Washer, 8mm", "qty": 100, "unit_price": 9.4}
  ]
}

result = score(prediction, ground_truth, schema)
result["accuracy"]    # 0.5294 -- matched addresses / addresses either document used
result["precision"]   # 0.5625
result["recall"]      # 0.8182

or from the command line.

bash

oeb score --pred pred.json --gt gold.json --schema schema.json

json

{
  "accuracy": 0.5294117647058824,
  "precision": 0.5625,
  "recall": 0.8181818181818182,
  "f1": 0.6666666666666666,
  "total": 17,
  "matched": 9,
  "misread": 1,
  "unfound": 1,
  "fabricated": 1,
  "invented_item": 4,
  "invented_field": 1,
  "asserted": 16,
  "addresses_found": 0.5882352941176471,
  "addresses_read_right": 0.9,
  "gt_rows": 2,
  "pred_rows": 3,
  "matched_rows": 2,
  "matching_exact": true,
  "approximated": [],
  "skipped_open_maps": []
}

Return with verdicts to dive deeper into exact places where the model failed.

python


result = score(prediction, ground_truth, schema, verdicts=True)
print(result["verdicts"][0]) # print the first verdict
# {
#       "address": [["k", "currency"]],
#       "gold_raw": null,
#       "pred_raw": "USD",
#       "gold_canon": null,
#       "pred_canon": "usd",
#       "verdict": "invented_field"
#     }

From the cli.

bash

oeb score --pred pred.json --gt gold.json --schema schema.json --verdicts

On the example above, printing one line each:

{
  "accuracy": 0.5294117647058824,
  "precision": 0.5625,
  "recall": 0.8181818181818182,
  "f1": 0.6666666666666666,
  ...
  "verdicts": [
    {
      "address": [
        [
          "k",
          "currency"
        ]
      ],
      "gold_raw": null,
      "pred_raw": "USD",
      "gold_canon": null,
      "pred_canon": "usd",
      "verdict": "invented_field"
    },
    ...truncated for display

What the metric does

[image]

—Normalize document;
—Flatten prediction and gold JSON dictionary to addresses mapped to their scalar values;
—Normalize scalar values of the flattened addresses; and
—For each array that appears, Hungarian match (recursively for nested arrays) based on array element content to align ambiguous predicted and gold addresses (there may unmatched predicted addresses — false positives, and unmatched gold addresses — false negatives).

Full specification: `METRIC_SPEC.md`.

Predict

Predict using our provider harnesses.

bash

uv pip install 'omni-extract-bench[harness]'

python

from omni_extract_bench.harness import predict

record = predict("datalab", "invoice.pdf", schema)
record["result"]
record["raw"]
record["cost"]

Score directly from the predictions.

python

from omni_extract_bench import score
score(record["result"], gold, schema)

You can also use the cli:

bash

oeb predict --provider datalab --doc invoice.pdf --schema schema.json

License

CC BY 4.0. The full texts are in licenses/.

suite	documents	upstream licence	credit to
`extractbench`	329	Apache 2.0	upstream authors
`internal`	202	Apache 2.0	Datalab — synthetic, generated for this benchmark
`longarray`	42	CC BY 4.0	Extend AI — cite LongArray-Extract, not this assembly
`micro1`	47	MIT	Micro1

The scorer itself is separate and Apache 2.0.

Citation

bibtex

@misc{omni_extract_bench,
  title  = {Omni Extract Bench},
  author = {Datalab},
  year   = {2026},
  url    = {https://github.com/datalab-to/omni_extract_bench}
}