Eamin-sust/BanglaEng-SynCorpus

BanglaEng-SynCorpus

Dataset Summary

BanglaEng-SynCorpus is a large-scale synthetic Bangla–English parallel corpus designed to support research in Neural Machine Translation (NMT) and other Bangla–English bilingual NLP tasks. The corpus is generated using linguistically validated sentence templates combined with topic-wise curated vocabularies, covering all 12 English/Bangla tense structures.

Due to extreme scale (trillions of possible sentence pairs), the dataset is released in multiple representative subsets, packaged as compressed .tar archives.

This dataset is particularly suitable for:

• —Low-resource Bangla–English NMT
• —Controlled synthetic data experiments
• —Curriculum learning and scaling studies
• —Sentence-structure-aware translation models

Supported Tasks

• —Machine Translation (Bangla ↔ English)
• —Text-to-text generation
• —Controlled synthetic data modeling
• —Linguistic and grammatical analysis

Languages

• —Bangla (bn)
• —English (en)

Dataset Structure

The repository contains tar-archived datasets generated by selecting different numbers of words per topic across all sentence structures.

Available Subsets

Each folder contains one or more .parquet files due to size constraints.

Data Generation Methodology

Vocabulary

• —3,990 parallel Bangla–English words
• —26 semantic categories, including:
• —Names, relations (with gender)
• —Animals, food, fruits, places, professions
• —Countries, cities, activities, objects, etc.

Sentence Structures

• —9,648 validated parallel sentence templates
• —Coverage of:
• —All 12 tense forms
• —Positive and negative sentences
• —Templates use category tags (e.g., <name>, <place/country/city>)

Generation Process

1. 1.Topic-wise word selection
2. 2.Template-based sentence expansion
3. 3.Automated grammatical validation
4. 4.Manual expert review of structures
5. 5.Duplicate and language-mixing checks

The full combinatorial corpus exceeds 2.9 trillion sentence pairs and is not fully stored due to storage limitations.

Data Format

The dataset is distributed as compressed .tar archives due to its large size. Each .tar file contains one or more Apache Parquet (`.parquet`) files.

Parquet Schema

Each Parquet file consists of two columns:

Each row represents a single Bangla–English parallel sentence pair.

File Organization

• —Subsets are organized based on the number of words selected per topic:
• —BanglaEng-SynCorpus-10Words/
• —BanglaEng-SynCorpus-20Words/
• —BanglaEng-SynCorpus-50Words/
• —BanglaEng-SynCorpus-80Words/
• —Large subsets are split into multiple .parquet files for easier storage and transfer.
• —All Parquet files inside a subset follow the same schema.

Accessing the Data

Users must first extract the .tar archives and then load the Parquet files using standard data-processing libraries such as PyArrow, pandas, or Apache Spark.

Intended Use

This dataset is intended for research and academic use, including:

• —Training and evaluating Bangla–English NMT models
• —Studying synthetic data scaling effects
• —Grammar-aware translation experiments

⚠️ Not recommended as a replacement for fully natural parallel corpora in real-world production systems without fine-tuning on human-translated data.

Limitations

• —Sentences are synthetically generated
• —Limited to simple sentence structures
• —Does not include discourse-level context
• —Vocabulary-driven, not frequency-driven

Ethical Considerations

• —No personal or sensitive data
• —No scraped or copyrighted text
• —Fully synthetic and template-generated

Citation

If you use this dataset, please cite:

@techreport{rahman2025banglaengsyncorpus,
  title={Developing a Synthetic Bangla-English Parallel Corpus for Neural Machine Translation},
  author={Rahman, Md. Eamin and Selim, Mohammad Reza},
  institution={Shahjalal University of Science and Technology},
  year={2025}
}

Authors & Contributors

Md. Eamin Rahman Assistant Professor, Dept. of CSE, SUST Hugging Face: Eamin-sust

Dr. Mohammad Reza Selim Professor, Dept. of CSE, SUST

License

This dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Acknowledgements

This dataset was developed at Shahjalal University of Science and Technology (SUST) under the SUST Research Centre Project (ID: AS/2024/1/25), titled “Developing a Synthetic Bangla–English Parallel Corpus for Neural Machine Translation.”

We sincerely thank all research assistants and annotators who contributed to the creation and validation of the corpus.

Contact

For questions or collaboration inquiries, please open an issue on the Hugging Face repository.