Davidsamuel101/ebible_local_ind_corpus

eBible Indonesian Local Language Corpus

This dataset contains parallel Bible translations between Indonesian and various local languages from Indonesia, particularly from Eastern Indonesia regions.

Dataset Description

This dataset is created from the eBible corpus, containing verse-aligned translations between Indonesian and multiple local languages. The dataset properly handles verse ranges where multiple verses are combined in the translation.

Available Language Pairs

The dataset contains the following language pairs as different configurations/subsets:

Total examples across all configurations: 99,713

Dataset Structure

Each configuration contains a 'train' split with the following features:

• —source_text: Text in the source language (Indonesian)
• —target_text: Text in the target language
• —source_lang: Source language code (always 'ind')
• —target_lang: Target language code
• —verse: Bible verse reference (e.g., "GEN 1:1" or "GEN 1:14-15" for verse ranges)

Usage

Loading a specific language pair

from datasets import load_dataset

# Load a specific language pair (configuration)
dataset = load_dataset("Davidsamuel101/ebible_local_ind_corpus", "ind_aaz")
train_data = dataset['train']

# Example: access first example
print(train_data[0])

Loading multiple configurations

from datasets import load_dataset

# Load specific configurations
target_langs = ["aaz", "ptu", "nfa", "heg", "lex", "row", "llg", "rgu", "txq", "tet", "wrs"]
for lang in target_langs:
    dataset = load_dataset("Davidsamuel101/ebible_local_ind_corpus", f"ind_{lang}")
    print(f"{lang}: {len(dataset['train'])} examples")

Using for machine translation

from datasets import load_dataset
from transformers import pipeline

# Load Indonesian to Tetun dataset
dataset = load_dataset("Davidsamuel101/ebible_local_ind_corpus", "ind_tet")['train']

# Example: prepare for training
def preprocess_function(examples):
    inputs = [ex for ex in examples["source_text"]]
    targets = [ex for ex in examples["target_text"]]
    return {"translation": [{"ind": i, "tet": t} for i, t in zip(inputs, targets)]}

# Apply preprocessing
processed_dataset = dataset.map(preprocess_function, batched=True)

Data Processing Notes

The dataset was created with special handling for:

• —Verse ranges: Marked with <range> tokens in the original corpus
• —Empty verses: Automatic skipping of verses where either source or target text is blank
• —Verse concatenation: Proper concatenation of source verses when targets combine multiple verses

Languages

Source Language

• —Indonesian (ind): Bahasa Indonesia, the national language of Indonesia

Target Languages

• —Amarasi (aaz): Uab Meto language spoken in West Timor
• —Helong (heg): Austronesian language of West Timor
• —Luang (lex): Language spoken in the Luang islands
• —Lole (llg): Language of Central Rote
• —Dhao (nfa): Language of Ndao island
• —Bambam (ptu): Language of Central Sulawesi
• —Rikou (rgu): Language of Central Rote
• —Dela-Oenale (row): Language of West Rote
• —Tetun (tet): Language of East Timor
• —Tii (txq): Language of West Rote
• —Waris (wrs): Papuan language

Citation

If you use this dataset, please cite:

@misc{ebible_indonesian_2024,
  title={eBible Indonesian Local Language Corpus},
  author={Dataset created from eBible corpus},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/Davidsamuel101/ebible_local_ind_corpus}
}

License

Please refer to the original eBible corpus license terms. The texts are from Bible translations which may have their own usage terms.

Acknowledgments

This dataset is derived from the eBible corpus. We acknowledge the translators and communities who created these Bible translations in local languages.