davanstrien/encyclopaedia_britannica_illustrated-lighton-ocr-32k-test

Document OCR using LightOnOCR-0.9B-32k-1025

This dataset contains OCR results from images in NationalLibraryOfScotland/Britain-and-UK-Handbooks-Dataset using LightOnOCR, a fast and compact 1B OCR model.

Processing Details

• —Source Dataset: NationalLibraryOfScotland/Britain-and-UK-Handbooks-Dataset
• —Model: lightonai/LightOnOCR-0.9B-32k-1025
• —Vocabulary Size: 32k tokens
• —Number of Samples: 100
• —Processing Time: 3.7 min
• —Processing Date: 2025-10-23 17:51 UTC

Configuration

• —Image Column: image
• —Output Column: markdown
• —Dataset Split: train
• —Batch Size: 32
• —Target Image Size: 1288px (longest dimension)
• —Max Model Length: 8,192 tokens
• —Max Output Tokens: 6,500
• —Temperature: 0.2
• —Top P: 0.9
• —GPU Memory Utilization: 80.0%

Model Information

LightOnOCR is a fast, compact OCR model that excels at:

• —⚡ Production Speed - 5.71 pages/second on H100 GPU
• —🎯 Compact Size - Only 1B parameters
• —📐 LaTeX formulas - Mathematical notation in LaTeX format
• —📊 Tables - Extracted and formatted as markdown
• —📝 Document structure - Hierarchy and layout preservation
• —🌍 Multilingual - Optimized for European languages
• —🔤 Flexible vocabulary - 151k/32k/16k token variants

Vocabulary Variants

• —151k tokens: Full vocabulary, supports all languages
• —32k tokens: European languages optimized (~12% faster decoding)
• —16k tokens: European languages optimized (~12% faster decoding)

Dataset Structure

The dataset contains all original columns plus:

• —markdown: The extracted text in markdown format with LaTeX formulas
• —inference_info: JSON list tracking all OCR models applied to this dataset

Usage

from datasets import load_dataset
import json

# Load the dataset
dataset = load_dataset("{output_dataset_id}", split="train")

# Access the markdown text
for example in dataset:
    print(example["markdown"])
    break

# View all OCR models applied to this dataset
inference_info = json.loads(dataset[0]["inference_info"])
for info in inference_info:
    print(f"Column: {info['column_name']} - Model: {info['model_id']}")

Reproduction

This dataset was generated using the uv-scripts/ocr LightOnOCR script:

uv run https://huggingface.co/datasets/uv-scripts/ocr/raw/main/lighton-ocr.py \
    NationalLibraryOfScotland/Britain-and-UK-Handbooks-Dataset \
    <output-dataset> \
    --vocab-size 32k \
    --image-column image \
    --batch-size 32

Performance

• —Processing Speed: ~0.45 images/second
• —Benchmark Score: 76.1% overall (across diverse document types)
• —Optimization: Native resolution ViT + lightweight decoder

Generated with 🤖 UV Scripts