acomquest/sanskrit-ocr-post-correction

Sanskrit OCR Post-Correction Dataset

Dataset Description

• —Homepage: https://github.com/ayushbits/pe-ocr-sanskrit
• —Repository: https://github.com/ayushbits/pe-ocr-sanskrit
• —Paper: A Benchmark and Dataset for Post-OCR text correction in Sanskrit

Dataset Summary

This dataset provides a benchmark for post-OCR text correction in Sanskrit. It contains manually post-edited OCR data from classical Sanskrit texts written in Devanagari script. The dataset is designed to help develop and evaluate models for correcting errors introduced during the OCR process of Sanskrit documents.

Supported Tasks and Leaderboards

• —Text Correction: The primary task is to correct OCR errors in Sanskrit text
• —Post-Editing: Can be used for training post-editing models for OCR outputs
• —Text Generation: Can be framed as a sequence-to-sequence text generation task

Languages

• —Sanskrit (Devanagari script)

Dataset Structure

The dataset is stored in efficient Parquet format for faster loading and processing.

Data Instances

Each instance contains an OCR output text and its manually corrected version:

{
  "input_text": "ज्योतिष्ठोमेन स्वर्गकामो यजेतेति वाक्ये स्वर्गका-मपदश्रवणेन तस्याधिकारविधित्वनिर्णयादिति भावः ।",
  "target_text": "ज्योतिट्षोमेन स्वर्गकामो यजेतेति वाक्ये स्वर्गका- मपदश्रवणेन तस्याधिकारविधित्वनिर्णयादिति भावः ।"
}

Data Fields

• —input_text (string): The OCR output text containing errors
• —target_text (string): The manually corrected ground truth text

Data Splits

The dataset contains four splits:

The ood_test split contains out-of-domain test examples as described in Section 4.1 of the paper.

Dataset Creation

Curation Rationale

This dataset was created to address the lack of resources for post-OCR correction in Sanskrit, a low-resource language with significant historical and cultural texts that need digitization.

Source Data

Initial Data Collection and Normalization

The dataset is derived from OCR outputs of classical Sanskrit texts including:

• —BHS: Brahmasutra Bhashyam
• —GG: Grahalaghava of Ganesh Daivajna
• —GOS: Goladhyaya

Refer paper for complete list of texts.

The OCR outputs were manually corrected by Sanskrit experts to create the ground truth.

Personal and Sensitive Information

The dataset contains classical Sanskrit texts and does not include any personal or sensitive information.

Considerations for Using the Data

Social Impact of Dataset

This dataset can help preserve and digitize Sanskrit literature by improving OCR accuracy for Sanskrit texts.

Discussion of Biases

The dataset is focused on classical Sanskrit texts and may not generalize well to modern Sanskrit usage or informal texts.

Other Known Limitations

• —Limited to Devanagari script
• —Focused on specific classical texts
• —May not cover all types of OCR errors

Additional Information

Dataset Curators

• —Ayush Maheshwari
• —Nikhil Singh
• —Amrith Krishna
• —Ganesh Ramakrishnan

Licensing Information

MIT License

Citation Information

@inproceedings{maheshwari2022benchmark,
  title={A Benchmark and Dataset for Post-OCR text correction in Sanskrit},
  author={Maheshwari, Ayush and Singh, Nikhil and Krishna, Amrith and Ramakrishnan, Ganesh},
  booktitle={Findings of the Association for Computational Linguistics: EMNLP 2022},
  pages={6258--6265},
  year={2022}
}