FormosanBank/formosan-mt

FormosanBank Machine Translation

Public parallel corpora for 15 Indigenous Formosan languages aligned with English and Mandarin Chinese. This release uses canonical MT-standardized Formosan text, excludes Formosan-Taiwan-Bible-Society-Bibles, and keeps DeepL pivot translations in training only.

Commercial AI use is prohibited without prior written permission. See the FormosanBank Terms of Use.

Release Summary

Human pairs in every language use 70% train, 10% validation, and 20% test for Formosan-English, and 85% train, 5% validation, and 10% test for Formosan-Chinese. Evaluation contains only eligible human sentence references. Synthetic pivots and short entries are added to training, so final augmented-corpus percentages are more train-heavy.

Languages

Schema

The two main files preserve the established nine-column format: id, source_lang, target_lang, source_sentence, target_sentence, lang_code, dialect, source, and row-level split.

from datasets import DatasetDict, load_dataset

rows = load_dataset("FormosanBank/formosan-mt", "formosan-en", split="train")
dataset = DatasetDict({
    split: rows.filter(lambda row: row["split"] == split)
    for split in ("train", "validate", "test")
})

Use formosan-zh for Chinese. Reverse-direction training can swap the sentence columns. Files under provenance/ map release IDs to source commits and XML records and include independent validation and TAME-MT reports.

Split Quality

The builder groups exact normalized pairs and punctuation skeletons, blocks one-edit conflicts, and excludes pair exposure at character 3-5 gram Jaccard similarity 0.95 or above. These are row-level hard splits, not document-held-out splits.

TM scores measure nearest-neighbor translation-memory retrieval, not model quality. All four directions have zero exact overlap and zero source, target, or pair exposure at 0.95.

Provenance

• —Public FormosanBank commit: 3a3c47c220520113f747e6a2d441494000e13c4b
• —FormosanBank QC commit: acc3ec9f3137a59b9661e446a750ee1606720394
• —Toolkit commit: 573488b215ee1cf3701519a5bcd1010e8f0c6548
• —MT standardization: formosan-mt-standard-v3
• —Build completed: 2026-08-24

Artifact hashes and per-language counts are recorded in provenance/release_metadata.json and SHA256SUMS.

Limitations

• —Sources vary in dialect, genre, translation style, and transcription quality.
• —Some human references retain source or linguistic annotations.
• —The English training corpus contains substantial synthetic augmentation.
• —Similarity controls do not prove semantic or document independence.
• —Automatic metrics do not replace evaluation by fluent speakers.

Citation

@misc{formosanbank_mt_public_v3,
  title        = {FormosanBank Machine Translation Public Corpus},
  author       = {FormosanBank contributors},
  year         = {2026},
  howpublished = {https://huggingface.co/datasets/FormosanBank/formosan-mt}
}

See the Terms of Use and AI Use Addendum.