Datasetpublic

eemberda/english-ceb-bible-prompt

LLM Benchmark for English-Cebuano Translation This dataset contains parallel sentences of English and Cebuano extracted from the Bible corpus available at https://github.com/christos-c/bible-corpus. The dataset is formatted for use in training machine translation models, particularly with the Transformers library from Hugging Face. Usage This dataset can be used to evaluate the performance of Large Language Models for English-Cebuano machine translation using… See the full description on the dataset page: https://huggingface.co/datasets/eemberda/english-ceb-bible-prompt.

sourceHugging Facecc-by-4.0updated 2y agoView on Hugging Face

0likes20downloads

Dataset Card

LLM Benchmark for English-Cebuano Translation

This dataset contains parallel sentences of English and Cebuano extracted from the Bible corpus available at https://github.com/christos-c/bible-corpus. The dataset is formatted for use in training machine translation models, particularly with the Transformers library from Hugging Face.

Dataset Description

—Source: Bible corpus (https://github.com/christos-c/bible-corpus)
—Languages: English (en) and Cebuano (ceb)
—Format: CSV files with two columns: english and cebuano.
—Extraction: Sentences were extracted from the XML files inside the <seg> tags, and matched by their id attribute.
—Purpose: To create a benchmark for English-Cebuano machine translation in Large Language Models.

Usage

This dataset can be used to evaluate the performance of Large Language Models for English-Cebuano machine translation using libraries like Transformers.

Example (Loading with Pandas):

python

import pandas as pd

train_df = pd.read_csv("english-ceb-bible-prompt.csv")
print(train_df.head())

Example (Loading with datsets):

python

from datasets import load_dataset

dataset = load_dataset("eemberda/english-ceb-bible-prompt")

Notes

The quality of the sentence alignment may vary. The dataset is intended for research and educational purposes. Please cite the original Bible corpus if you use this dataset in your work.

Citation

If you use this dataset in your research, please include the original Bible corpus in your citations:

@misc{bible-corpus,
  author = {Christos Christodoulopoulos},
  title = {Bible Corpus},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{[https://github.com/christos-c/bible-corpus](https://github.com/christos-c/bible-corpus)}}
}