emanuelaboros/pleias-post-ocr-correction-chonkie-aligned-fr

PleIAs Post-OCR Correction — Chonkie-Aligned Semantic Chunks

This dataset is a semantically chunked and span-aligned derivative of `PleIAs/Post-OCR-Correction`.

Each record contains:

• —an OCR hypothesis chunk from the original text field;
• —a corresponding post-OCR correction output chunk from the corrected_text field;
• —metadata inherited from the PleIAs dataset;
• —character spans linking each chunk back to the original source document;
• —alignment diagnostics produced during filtering.

Important note

The corrected_text field in the original PleIAs dataset is an experimental multilingual post-OCR correction output. It should not be treated as manually verified ground truth.

For this reason, this derivative dataset stores:

ocr_hypothesis = original PleIAs text
ground_truth = PleIAs corrected_text

Filtering

This version removes suspicious alignment cases detected by automatic heuristics.

• —Source file: pleias_french_sample.jsonl
• —Clean file: pleias_french_sample.clean.jsonl
• —Total records inspected: 7,435
• —Kept records: 4,835
• —Removed suspicious records: 2,600
• —Removed share: 34.97%

Suspicious cases were detected using:

• —empty OCR or corrected chunks;
• —very short chunks;
• —extreme OCR/corrected length ratios;
• —low character-level similarity;
• —high CER-like edit distance.

The removed cases and CSV diagnostics are included in the diagnostics/ folder.

Format

Each line in data.jsonl is one JSON record with this structure:

{
  "document_metadata": {},
  "ground_truth": {},
  "ocr_hypothesis": {},
  "ground_truth": {}
}

Recommended use

This dataset can be used for experiments in OCR post-correction, noisy text normalization, historical document processing, and evaluation of post-OCR correction systems.

Because the correction target is synthetic, it is better suited for weak supervision, pretraining, filtering experiments, or diagnostic analysis than for gold-standard benchmark evaluation.