gt-free-ocr-metrics/ocr-render-compare-omnidocbench

OmniDocBench render-and-compare pairs, four OCR engines

Render-and-compare pairs for the 1,355 pages of OmniDocBench v1.5 from four OCR engines: Qwen3.5-122B-A10B (qwen35), MinerU-2.5 (mineru25), PaddleOCR-VL (paddleocr_vl), DeepSeek-OCR (deepseek_ocr). The OCR output of each page is scored in five variants: ocr_text (text elements only), ocr_formula (formulas only), ocr_table (tables only), ocr_all (all three, with regions of other types masked on the original) and ocr_all_no_mask (all three, nothing masked). Each row holds the masked original, the rendered OCR output, the OCR output itself, the reference quality from OmniDocBench's official evaluator, and reference-free scores: the fitted composite (Qwen3.5 only, held out), frozen ColPali and ink-confusion F1.

Part of a set of datasets released with a benchmark for evaluating document OCR without ground truth by render-and-compare: the OCR output is drawn back into a page image and compared with the original. All datasets: gt-free-ocr-metrics/ocr-render-compare. Code: https://anonymous.4open.science/r/GT-free-ocr-evaluation-804F/

Subsets

Each subset has one split, test.

The nine subsets

qwen35, mineru25, paddleocr_vl, deepseek_ocr: one OCR engine each

One row per page and variant for the engine named. The variants of a page share one transcript, the engine's full output for the page (ocr_html and the element JSON columns); the variant decides which of its elements are evaluated, which regions are whitened on the original and which elements are drawn in the render.

A row exists wherever the engine emitted an element of the variant's type, so the counts differ by engine. reference is empty on pages the official evaluator does not score for that variant; the paper's correlations use the rows that have one (for qwen35: 1,290 / 200 / 351 / 1,344 / 1,344 pages).

Rows carrying each score, per variant in the order ocr_text / ocr_formula / ocr_table / ocr_all / ocr_all_no_mask:

• —reference: OmniDocBench's own end-to-end evaluator, as in the variant table: all five variants for Qwen3.5; ocr_text, ocr_all and ocr_all_no_mask for the other engines.
• —cdm: Character Detection Matching of the page's formulas, the mean over the formula pairs the evaluator matched: the benchmark's own formula metric, used as a second formula reference (Qwen3.5 ocr_formula).
• —composite: the paper's fitted composite, held out: its weights are fitted on four document-grouped folds and applied to the fifth (Qwen3.5).
• —colpali: frozen ColPali v1.2: the 5th percentile of the per-element similarities, or the whole-page similarity where a page has no element of the variant's type.
• —ink_f1: ink-confusion F1 between the binarized masked original and render, with a tolerance of half the line pitch measured on the unmasked page.

pages

The 1,355 unmasked OmniDocBench page images, stored as distributed (981 JPEG, 374 PNG), with the benchmark's page attributes: category (9 document types: PPT2PDF, academicliterature, book, colorfultextbook, exampaper, magazine, newspaper, note, researchreport), language (enchmixed, english, simplifiedchinese), `layout` (1andmorecolumn, doublecolumn, otherlayout, singlecolumn, threecolumn) and special_issue (for example watermark, colorful_backgroud in the benchmark's own spelling, or table attributes; empty when none). Join with the other subsets on page_id. Ink-confusion F1 takes its line pitch from these images.

catalog_scores

580,860 rows: every page and variant of Qwen3.5's renders scored by each of the 126 cataloged configurations (the 125 implementations that produced full per-variant scores, and the SSIM+MSE+LPIPS baseline). Each configuration writes its own score to one of two columns; the other carries what the configuration it was built on computed. The own score is clip_cosine, except for baseline and the configurations that replace or extend a term of the SSIM+MSE+LPIPS blend (P1_024_deepwsd_wasserstein_metric, P1_025_ms_swd_table_4th_term, P1_066_hashes, P1_082_st_lpips, P1_120_adists_additive, P1_120b_adists_variant_conditional, P1_121_dists_baseline, P2_060k_docsim_mc_4thterm_all), whose own score is multi_composite. Join with qwen35 on page_id and variant and correlate with reference to reproduce the catalog; each configuration is specified in methods/<method_id>.yaml in the code.

qwen35_logprobs

1,355 rows, one per page: per_bbox_logprobs, a JSON object from Qwen3.5-122B-A10B's OCR pass with n_total_tokens, n_bboxes and bboxes, one entry per output element (bbox_id, div_class, data_bbox, n_tokens, and stats: logprob_mean, logprob_min, logprob_max, shannon_entropy_mean, shannon_entropy_max). The composite's entropy term and the self-uncertainty configurations read these.

qwen35_token_logprobs

1,355 rows, one per page: token_logprobs, the JSON record of Qwen3.5-122B-A10B's OCR pass (tokens: every generated token with its logprob and top_logprobs), and ocr_html, the transcript that pass produced. The per-element statistics above are computed from these. scripts/extract_hf_dataset.py in the code writes them to data/ocr_logprobs/<page_id>/, where the log-probability configurations read them.

docsim_triplets

20,280 rows, one per triplet, in the order of the training manifest: anchor_path, the masked original of a page; positive_path, the render of the same page; negative_path, the render of another page, with their pages and variants and *_ed, the text edit distance of each page's transcript to the ground truth (0 for anchors and positives, at least 0.415 for negatives). Paths name images of the qwen35 subset as <variant>/<page_id>/<file>, the layout scripts/extract_hf_dataset.py writes under data/omnidocbench/. The DocSim head, shipped in the code repository (models/docsim_lora/), was trained on these.

Loading

from datasets import load_dataset

ds = load_dataset("gt-free-ocr-metrics/ocr-render-compare-omnidocbench", "qwen35", split="test")

Columns

• —page_id: page identifier
• —variant: which output elements are scored
• —category: document category
• —language: page language
• —layout: OmniDocBench layout attribute
• —reference: ground-truth-based quality from OmniDocBench's official evaluator (higher is better)
• —reference_metric: which evaluator field reference is
• —cdm: CDM of the formulas (Qwen3.5, ocr_formula)
• —composite: the paper's fitted composite, held out (Qwen3.5 only)
• —colpali: frozen ColPali v1.2, per-element 5th percentile with whole-page fallback
• —ink_f1: ink-confusion F1
• —masked_original: original page, whitened over detected elements the variant does not evaluate (PNG)
• —reconstructed: evaluated elements drawn at their boxes on a white canvas of the page's size (PNG)
• —ocr_html: OCR output, HTML with element boxes
• —ocr_elements: parsed text elements (JSON)
• —ocr_formula_elements: parsed formula elements (JSON)
• —ocr_table_elements: parsed table elements (JSON)
• —special_issue: OmniDocBench special-issue attribute (e.g. fuzzy scan, watermark)
• —page_image: unmasked page image
• —method_id: cataloged configuration
• —clip_cosine: score slot holding each configuration's own signal, except the baseline and the seven configurations that extend it (see METRIC_OF in buildresultsdf.py of the code)
• —multi_composite: SSIM+MSE+LPIPS score slot, the own signal of the baseline and the seven configurations that extend it
• —per_bbox_logprobs: per output element: token count and log-probability / entropy statistics (JSON)
• —token_logprobs: every generated token with its log-probability and top alternatives (JSON)
• —anchor_path: masked original of a page (<variant>/<page_id>/masked_original.png)
• —positive_path: render of the same page
• —negative_path: render of another page
• —anchor_ed: text edit distance of the anchor page's transcript to the ground truth
• —positive_ed: the same for the positive
• —negative_ed: the same for the negative
• —anchor_page: page of the anchor
• —positive_page: page of the positive
• —negative_page: page of the negative
• —anchor_variant: variant of the anchor
• —positive_variant: variant of the positive
• —negative_variant: variant of the negative
• —kind: triplet type

How the images were made

For a variant, masked_original is the page image whitened over every element the engine detected whose type the variant does not evaluate (images included); what the engine did not detect is never masked and stays on the page. reconstructed draws each element of the evaluated types at its box on a white canvas of the page's size (text with estimated font size, weight and alignment; formulas through mathtext or pdflatex; tables from their HTML grid). A reference-free metric compares the two. Both are the PNG files the paper scored, stored byte for byte (no re-encoding).

Intended use

Measuring how well reference-free OCR quality metrics track ground-truth-based quality, and developing new ones. The renders are model outputs, not ground truth; OCR text must not be used to redistribute the source documents.

Source and license

Derived from OmniDocBench, which is for research use only and not for commercial use. Released under CC BY-NC 4.0.