rejauldu/bengali-wikipedia

📚 Bengali Wikipedia Language Modeling Dataset (For GPT-2 Training)

📝 Dataset Summary

This dataset contains a large Bengali text corpus collected from Bengali Wikipedia. It is cleaned, sentence-segmented, and formatted for next-token prediction language modeling tasks such as GPT-2 training.

It includes train and validation splits, suitable for transformer-based Bengali language models.

📊 Dataset Details

🔧 Preprocessing

• —Non-article/irrelevant pages removed (redirects, templates, etc.)
• —Special markup cleaned (HTML/wiki tags)
• —Kept Unicode Bengali characters with character coverage 0.9995
• —Split into lines of clean sentences
• —Tokenizer-friendly format (no artificial spaces)

🔀 Dataset Splits

Example code to load splits:

from datasets import load_dataset

dataset = load_dataset("rejauldu/bengali-wikipedia")
print(dataset)
print(dataset['train'][0])

🧩 Tokenization Recommendation

Suitable for training with:

✅ SentencePiece — Unigram (recommended for pure Bengali) ✅ SentencePiece BPE — for mixed English-Bengali text ✅ GPT-2 Byte-Level BPE — if replicating original GPT-2 behavior

Example SP training config:

spm_train \
  --input=bn_corpus_train.txt \
  --model_prefix=bn_unigram \
  --vocab_size=50000 \
  --model_type=unigram \
  --character_coverage=0.9995 \
  --user_defined_symbols=<bos>,<eos>,<pad>

✅ Example Usage (for GPT-2 Fine-tuning)

from transformers import GPT2TokenizerFast, GPT2LMHeadModel, Trainer, TrainingArguments
from datasets import load_dataset

dataset = load_dataset("rejauldu/bengali-wikipedia")

tokenizer = GPT2TokenizerFast.from_pretrained("path/to/spiece/model")
tokenizer.pad_token = tokenizer.eos_token

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True, padding="max_length", max_length=512)

dataset = dataset.map(tokenize, batched=True, num_proc=4)
dataset.set_format(type="torch", columns=["input_ids", "attention_mask"])

model = GPT2LMHeadModel.from_pretrained("gpt2")
model.resize_token_embeddings(len(tokenizer))

📌 Ethical Considerations

• —Text from Wikipedia may contain biases, outdated info, or culturally sensitive entities.
• —Use responsibly; apply additional filtering for safety-critical deployments.

🏷️ Citation

If you use this dataset, please cite:

Wikipedia contributors. (CC BY-SA 3.0).
Bengali Wikipedia dump.
https://bn.wikipedia.org/