sk-community/romanized_bangla

romanized_bangla — Dataset Card

Repository / id: sk-community/romanized_bangla Derived from: wikimedia/wikipedia subset 20231101.bn (Bangla Wikipedia dump)

1. Short description

A romanized (Latin-script) version of Bangla Wikipedia text derived from the wikimedia/wikipedia dataset (subset 20231101.bn). The original dataset contained whole-article entries; you split article paragraphs into individual rows and transliterated Bangla text into a romanized representation using a Python script. The processed dataset contains 975,215 rows (paragraph-level documents) in romanized Bangla.

This dataset is intended for tasks such as: language modeling, back-transliteration, transliteration normalization, pretraining/finetuning language models that support Romanized Bangla, and dataset augmentation for downstream Bangla NLP tasks.

2. Dataset summary

• —Original source: wikimedia/wikipedia — subset 20231101.bn (Bangla).
• —Original size (articles): ~143,069 rows (article-level).
• —Processed size (paragraph-level, romanized): 975,215 rows (as uploaded to sk-community/romanized_bangla).
• —Languages: Romanized Bangla (Latin script).
• —Modality: Text ---

• —Transliteration script: https://github.com/sk-research-community/multilingual-transliterator-llm-training/blob/main/src/rulebasedbengali_transliterate.py

3. Recommended usage

• —Tasks: language modeling, transliteration/back-transliteration, normalization, data augmentation, pretraining/finetuning transformer models that should accept romanized Bangla.
• —Load example:

from datasets import load_dataset

ds = load_dataset("sk-community/romanized_bangla", split="train")
print(next(iter(ds))["text"])  # prints a romanized paragraph

• —Example preprocessing for training: tokenization with an appropriate tokenizer (Byte-Pair, SentencePiece, or multilingual tokenizers such as XLM-R based tokenizers) and typical ML preprocessing pipelines.

4. Train/validation/test splits

Currently the dataset is provided as a single train split. If you plan to publish splits for downstream tasks, consider adding train/validation/test splits and provide reproducible splitting code (e.g., deterministic by hashing id or using datasets traintestsplit).

5. Licensing and terms of use

This dataset is distributed under the MIT License. Please also check the licenses of the individual source datasets before use.

6. Citation

@article{gharami2025indotranslit,
  title={Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integration},
  author={Kanchon Gharami and Quazi Sarwar Muhtaseem and Deepti Gupta and Lavanya Elluri and Shafika Showkat Moni},
  year={2025}
}

7. Maintainers / Contact

• —Maintainer: sk-community ---