vichetkao/paragraph_translation_eng_kh

Machine Translation Dataset: Khmer to English

A machine translation dataset project that pairs Khmer document images with their corresponding English translations. This dataset is designed for training machine translation models to convert Khmer text (via OCR from images) to English text.

Project Overview

This project processes bilingual documents from BOP (Bank of PNG) Bulletins, creating a Khmer-English machine translation dataset. Each sample pairs a Khmer image (source) with its English text translation (target). The dataset is automatically split into training (80%) and testing (20%) sets for model development and evaluation.

Purpose: Build a machine translation model that takes Khmer document images as input and outputs English translation text.

Project Structure

./
├── data/                              # Generated dataset
│   ├── image_pairs_train.parquet      # Training (80%)
│   └── image_pairs_test.parquet       # Test (20%)
└── README.md                          # This file

Features

• —Image-Text Pairing: Matches Khmer images with English translations
• —Bilingual Documents: Sources from official Bank of PNG bulletins
• —Automatic Splitting: 80/20 train-test split for ML model training
• —Data Validation: Ensures image integrity and proper pairing
• —Efficient Format: Saves in Parquet for fast loading and processing
• —Batch Processing: Handles multiple document folders automatically

Installation

Requirements

• —Python 3.7+
• —pandas
• —scikit-learn
• —Pillow (PIL)
• —pyarrow

Setup

pip install pandas scikit-learn pillow pyarrow

Usage

Processing and Creating Dataset

Run the Jupyter notebook match_and_save_pairs.ipynb to:

1. 1.Match Khmer images with English translations
2. 2.Validate image quality and data integrity
3. 3.Automatically split into train (80%) and test (20%)
4. 4.Save to Parquet files in data/ folder

Loading Dataset for Machine Translation

import pandas as pd
from PIL import Image
from io import BytesIO

# Load training dataset
df_train = pd.read_parquet("./data/image_pairs_train.parquet")

# Load test dataset
df_test = pd.read_parquet("./data/image_pairs_test.parquet")

# Access a sample
khmer_image_bytes = df_train.iloc[0]['kh_image']['bytes']
english_translation = df_train.iloc[0]['eng_image']['bytes']

# Convert Khmer image bytes to PIL Image for viewing
khmer_image = Image.open(BytesIO(khmer_image_bytes))
khmer_image.show()

print("English Translation:", english_translation)

Dataset Structure for ML Models

Each training sample contains:

• —Input: Khmer document image (source - contains Khmer text to be translated)
• —Output: English translation text (target - expected translation result)

Ideal for:

• —OCR + Translation: Extract Khmer text from image, then translate to English
• —Image-to-Text Translation: Direct visual machine translation
• —Vision-Language Models: Multi-modal translation systems

Dataset Format

Each sample in the Parquet files is structured for machine translation:

{
    'kh_image': {'bytes': <khmer_image_bytes>},      # Input: Khmer document image
    'eng_image': {'bytes': <english_text_bytes>}     # Output: English translation
}

• —image_pairs_train.parquet: 80% of samples (training set)
• —image_pairs_test.parquet: 20% of samples (test set)

Data Characteristics

• —Input (kh_image): Raw image bytes of Khmer documents from BOP bulletins
• —Output (eng_image): Corresponding English text translation
• —Format: Binary Parquet files for efficient storage and loading
• —Total Samples: Matched pairs split 80/20 for train/test

License

License: Kao Vichet (Custom License)

Data Source

Source: NBC (National Bank of Cambodia)

The image data is sourced from Bank of PNG official bulletins and publications provided by the National Bank of Cambodia.

Author

• —Kao Vichet
• —Project: Machine Translation Eng-Kh

Last Updated: March 2026