cloverx-id/xone-repository-parallel-en-id-corpus

We are currently developing new version of LMSE translation scoring model and processing additional data sources. We estimate the dataset will expand, with significantly improved quality.(Delayed..)

A score of 55% and above indicates high-quality translation pairs, even if the first version of the model we developed gave them such a score. We will try to release a newer model in the future with better quality and consistently fast scoring speeds, and release it to the public once we decide that this model is ready to be published.

Xone Parallel English-Indonesian Corpus

The Xone Parallel English-Indonesian Corpus is a large-scale parallel corpus containing over 70 million raw sentence pairs compiled from 11 translation datasets. This repository provides sanitized training pairs as well as a separate subset containing filtered entries for diagnostic transparency.

Subsets and Configurations

The dataset is organized into two configurations:

Column Schema

1. Subset default (Clean Data)

• —`no` (int64): Unique global row index.
• —`en` (string): Sanitized English text.
• —`id` (string): Sanitized Indonesian text.
• —`source` (string): Dataset provenance (e.g., opus100, opus_ccmatrix_en_id, paracrawl, opus_opensubtitles_en_id, quran_translation, flores, nusax_mt_parallel).
• —`domain` (string): Domain label (multi, subtitle, berita, keagamaan, artikel wikipedia).

2. Subset drop_by_filter (Rejected Data)

• —`no` (int64): Unique rejected row index.
• —`en` (string): English text.
• —`id` (string): Indonesian text.
• —`source` (string): Dataset provenance.
• —`domain` (string): Domain label.
• —`reason` (string): Filter rejection reason. Categories include:
• —REJECTED_SPAM: Detected online gambling or promotional spam.
• —REJECTED_MOJIBAKE: Corrupted UTF-8 text encoding (e.g., â, ’).
• —REJECTED_URL_OR_CODE: Contains raw web URLs (http://) or code snippets.
• —REJECTED_TOO_SHORT: Text length shorter than 2 characters after cleaning.
• —REJECTED_UNTRANSLATED_IDENTICAL: Identical English and Indonesian text for sentences exceeding 3 words.
• —REJECTED_INVALID_RATIO: Extreme word count ratio imbalance between English and Indonesian (outside 0.20 - 4.00 range).
• —REJECTED_DUPLICATE_HASH: Exact duplicate sentence pair detected via 64-bit Blake2b hashing.

Sanitization and Filtering Rules

• —Hyphen Normalization: Hyphens are strictly preserved when enclosed by alphanumeric characters (e.g., well-known, anak-anak). Leading, trailing, or standalone hyphens are removed.
• —Punctuation and Brackets: HTML entities are unescaped, HTML/subtitle formatting tags are stripped, and unbalanced parentheses or brackets are fixed via an automated bracket-balancing handler.
• —Corpus Richness Preservation: Proper nouns, brand names, location names, and parenthetical expressions are fully preserved to ensure dataset diversity for neural machine translation (NMT) and large language model (LLM) training.
• —Deduplication: 64-bit Blake2b hashing ensures zero exact duplicate sentence pairs in the default train subset.

Usage Example

from datasets import load_dataset

# Load Clean Training Data
dataset_clean = load_dataset("cloverx-id/xone-repository-parallel-en-id-corpus", name="default", split="train")

# Load Rejected Data for Diagnostics
dataset_dropped = load_dataset("cloverx-id/xone-repository-parallel-en-id-corpus", name="drop_by_filter", split="train")

# Streaming Mode for Memory-Constrained Environments
dataset_stream = load_dataset("cloverx-id/xone-repository-parallel-en-id-corpus", name="default", split="train", streaming=True)
for sample in dataset_stream:
    print(sample["en"], "->", sample["id"])
    break

Citation

If you use this dataset in your research or project, please cite it as follows:

@dataset{luminamoon2026xone_parallel_corpus,
  author       = {{Silver Moon (cloverxion)} and {Earl Pan (earlpan-id)}},
  organization = {Lumina Moon},
  title        = {{xone-repository-parallel-en-id-corpus}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/cloverx-id/xone-repository-parallel-en-id-corpus}},
  url          = {https://huggingface.co/datasets/cloverx-id/xone-repository-parallel-en-id-corpus},
  note         = {Hugging Face Dataset}
}