Datasetpublic

taresco/KarantaOCR-Bench

KarantaOCR - Bench KarantaOCR-Bench is a unit-test–style evaluation dataset, similar to olmOCR-bench. It consists of 70 PDF documents and 300 test cases spanning multiple document types. All the tests were manually verified by us. KarantaOCR-Bench is designed specifically to evaluate document text extraction for Documents with diacritics and special characters, covering a diverse range of document formats and languages commonly under-represented in existing OCR benchmarks.… See the full description on the dataset page: https://huggingface.co/datasets/taresco/KarantaOCR-Bench.

sourceHugging Faceodc-byupdated 8mo agoView on Hugging Face

0likes396downloads

Dataset Card

KarantaOCR - Bench

KarantaOCR-Bench is a unit-test–style evaluation dataset, similar to olmOCR-bench. It consists of 70 PDF documents and 300 test cases spanning multiple document types. All the tests were manually verified by us. KarantaOCR-Bench is designed specifically to evaluate document text extraction for Documents with diacritics and special characters, covering a diverse range of document formats and languages commonly under-represented in existing OCR benchmarks.

Data Schema

Below is a schema of each test case and what it means:

—pdf [str]: This is the name of the pdf file in the pdf directory
—page [str]: This is the page of the document for this test case
—id [str]: Unique ID for this test case
—type [str]: This is the test type; a description of each test type can be found in the next section
—max_diffs [int]: This is the maximum allowable difference in the OCR outputs and test case for it to pass
—text [str]: This is the text span that we search for in the OCR outputt.
—case_sensitive [bool]: This indicates whether the test case is case sensitive or not
—checked [str]: This indicates whether or not we have verified this test case manually
—url [str]: The original file name of the document
—before [str]: This only applies to text-order tests and this is the text span that should exist before
—after [str]:This only applies to text-order tests and this is the text span that should exist after

Test Case Types

KarantaOCR-Bench is designed to augment OlmoOCR test with an additional test category for Diacritics and Special Chars:

—Presence: Presence tests check whether a provided span of text exists in the OCR output. Matches are evaluated with a tolerance defined by the max_diffs field, allowing for limited character-level differences.

Results

JSONL File	KarantaOCR (3B)	RoLMOCR (7B)	NanoNetsOCR-2 (3B)	OLMOCR-2 (7B)
Diacritics & Special Characters	61.6	53.6	55.1	55.5

How to Run the Benchmark on your Model

We are working on a comprehensive guide on how to evaluate your models on this benchmark.

Coming Soon...

Citation Information

Coming soon ...