mehedihasanbijoy/BanglaPRCorpus

BanglaPRCorpus

A 1.48M-pair corpus for Bangla punctuation restoration — unpunctuated source sentences paired with their fully punctuated targets, labelled by how many punctuation marks were removed.

BanglaPRCorpus is the corpus introduced in *Advancing Bangla Punctuation Restoration by a Monolingual Transformer-Based Method and a Large-Scale Corpus* (Bijoy et al., EMNLP 2023 Workshop on Bangla Language Processing), alongside the Jatikarok model.

Each row is a (source, target) pair: the target is a well-punctuated Bangla sentence, and the source is the same sentence with nopr punctuation marks stripped out. Sentences are derived from the BanglaParaphrase corpus.

Quick start

from datasets import load_dataset

ds = load_dataset("mehedihasanbijoy/BanglaPRCorpus")

print(ds)
# DatasetDict({
#     train:      Dataset({features: ['source', 'target', 'nopr'], num_rows: 1125680})
#     validation: Dataset({features: ['source', 'target', 'nopr'], num_rows:   59242})
#     test:       Dataset({features: ['source', 'target', 'nopr'], num_rows:  296227})
# })

print(ds["train"][0])
# {'source': 'বিমানটি যখন মাটিতে নামার জন্য এয়ারপোর্টের কাছাকাছি আসছে, তখন ল্যান্ডিং গিয়ারের খোপের ঢাকনাটি খুলে যায়',
#  'target': 'বিমানটি যখন মাটিতে নামার জন্য এয়ারপোর্টের কাছাকাছি আসছে, তখন ল্যান্ডিং গিয়ারের খোপের ঢাকনাটি খুলে যায়।',
#  'nopr': 1}

Stream it instead of downloading the full 670 MB:

ds = load_dataset("mehedihasanbijoy/BanglaPRCorpus", split="train", streaming=True)
for row in ds.take(3):
    print(row["source"], "→", row["target"])

Dataset structure

Fields

Splits

nopr distribution

Removal count is heavily skewed toward 1 — most Bangla sentences carry only a sentence-final ।.

nopr is a difficulty axis: higher values mean more marks to restore in one sentence. Report per-`nopr` metrics — an aggregate score is dominated by the easy nopr=1 case, which is 63% of the corpus.

Sentence statistics

Punctuation inventory

16 marks are treated as punctuation by the corpus builder. Their frequency across all targets (3,180,611 marks total):

The builder's list also includes “ and ‘, but cleaning removes them before punctuation stripping, so they do not appear in targets.

Examples

How the splits were made

Splitting follows the convention in the Jatikarok reference implementation. Within each nopr group, the last 20% of rows in corpus order become test and the first 80% become train:

for idx in sorted(df['nopr'].unique()):
    temp_df = df[df['nopr'] == idx].reset_index(drop=True)
    x = int(len(temp_df) * .2)
    train = temp_df.iloc[:len(temp_df)-x, :]
    test  = temp_df.iloc[len(temp_df)-x:, :]

The split is stratified by nopr, and is a positional tail-slice rather than a random sample — there is no shuffling and no RNG, so it reproduces exactly. Validation is carved from the tail of each group's train portion by the same rule (5% of the remainder). Regenerate with the included make_splits.py.

Relationship to the published Jatikarok results

This release differs from the split in the paper's main.py in two ways, so it will not reproduce the published numbers exactly:

1. 1.All `nopr` values are included. The paper's loop is written for idx in range(10), which yields 0..9 — but nopr takes values 1..10. So nopr == 0 matched nothing and the 683 nopr == 10 rows were silently dropped. Those rows are included here.
2. 2.There is a validation split. The paper's code builds only train_df and test_df and evaluates on test directly. Carving out validation shrinks train from 80% to 76%.

The test set is otherwise the same tail-slice, so results are broadly comparable, but a strict reproduction of the paper's table requires its exact train set.

Example usage

Fine-tune a seq2seq model

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

ds = load_dataset("mehedihasanbijoy/BanglaPRCorpus")

model_name = "csebuetnlp/banglat5_banglaparaphrase"
tok = AutoTokenizer.from_pretrained(model_name, use_fast=False)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

def preprocess(batch):
    enc = tok(batch["source"], max_length=128, truncation=True)
    enc["labels"] = tok(batch["target"], max_length=128, truncation=True)["input_ids"]
    return enc

tokenized = ds.map(preprocess, batched=True, remove_columns=ds["train"].column_names)

The paper benchmarks three models this way: Jatikarok (Helsinki-NLP/opus-mt-NORTH_EU-NORTH_EU, MarianMT, with knowledge transferred from a Bangla GEC task), BanglaT5, and T5-Small.

Evaluate per nopr — the number that actually matters

test = load_dataset("mehedihasanbijoy/BanglaPRCorpus", split="test")

for n in range(1, 11):
    subset = test.filter(lambda r: r["nopr"] == n)
    preds = my_model.restore(subset["source"])
    em = sum(p == g for p, g in zip(preds, subset["target"])) / len(subset)
    print(f"nopr={n:>2}  n={len(subset):>6}  exact-match={em:.4f}")

The paper reports micro-averaged accuracy / precision / recall / F1 / F0.5 over whole decoded sentences. Because those are computed over full sentence strings with average='micro', they all collapse to the same value — exact sentence match. Per-punctuation F1 (below) is more informative.

Per-punctuation-mark F1

Exact match is harsh: one missed comma in a 40-word sentence scores zero. Scoring each mark separately shows which marks a model gets wrong.

from collections import Counter

PUNCS = set("।,!?;ঃ:-(){}[]")

def mark_positions(sent):
    """Map each punctuation mark to the index of the word it follows."""
    out = Counter()
    for widx, word in enumerate(sent.split()):
        for ch in word:
            if ch in PUNCS:
                out[(widx, ch)] += 1
    return out

tp = fp = fn = 0
for pred, gold in zip(preds, golds):
    p, g = mark_positions(pred), mark_positions(gold)
    tp += sum((p & g).values())
    fp += sum((p - g).values())
    fn += sum((g - p).values())

precision = tp / (tp + fp)
recall    = tp / (tp + fn)
f1        = 2 * precision * recall / (precision + recall)
print(f"P={precision:.4f}  R={recall:.4f}  F1={f1:.4f}")

Frame it as token classification instead

Predict, for each word, which punctuation mark follows it. This is the standard formulation in the punctuation-restoration literature and is far cheaper than generation.

PUNCS = set("।,!?;ঃ:-(){}[]")

def to_tags(row):
    """Align each source word with the mark that follows it in the target."""
    words, t, tags = row["source"].split(), row["target"].split(), []
    for i, w in enumerate(words):
        tag = "O"
        if i < len(t):
            trailing = [c for c in t[i] if c in PUNCS]
            if trailing:
                tag = trailing[-1]
        tags.append(tag)
    return {"words": words, "tags": tags}

tagged = ds["train"].map(to_tags)

Word counts can differ between source and target when a removed mark was a standalone token, so validate alignment before training on it.

Build a harder benchmark subset

nopr=1 is 63% of the corpus and mostly amounts to appending a ।. To test real punctuation reasoning, restrict to multi-mark sentences:

hard = test.filter(lambda r: r["nopr"] >= 3)
print(len(hard))  # 35,618

Few-shot prompting with an LLM

train = load_dataset("mehedihasanbijoy/BanglaPRCorpus", split="train")
shots = train.shuffle(seed=0).select(range(5))

prompt = "Restore the missing punctuation in the Bangla sentence.\n\n"
for s in shots:
    prompt += f"Input: {s['source']}\nOutput: {s['target']}\n\n"
prompt += f"Input: {test[0]['source']}\nOutput:"

Limitations and caveats

Please read these before reporting numbers.

• —`ঃ` (visarga) is treated as punctuation. It is a Bangla diacritic that occurs inside words, so stripping it corrupts the word rather than removing punctuation — e.g. নিঃসন্দেহে → নিসন্দেহে. It accounts for 21,632 marks (0.68%). Models must learn to repair intra-word visarga, which is arguably a different task from punctuation restoration.
• —`-` (hyphen) is likewise often word-internal, and at 8.43% of all marks it is not a small effect.
• —The split is positional, not random. Test is the tail of each nopr block in corpus order. If the underlying BanglaParaphrase corpus has any ordering structure (by source document or topic), train and test are not i.i.d.
• —A few duplicate pairs cross splits. The corpus has 120,494 duplicate (source, target) rows; 279 test rows (0.09%) have an identical pair in train, and 218 of 186,526 distinct test targets also appear as train targets. Small, but non-zero.
• —Target sentences are shared across `nopr` values. The same sentence appears with 1, 2, 3… marks removed. Rows for a given sentence therefore sit in different nopr blocks and may land in different splits — deduplicate on target if you want strict sentence-level separation.
• —Which marks get removed is random. The builder shuffles the punctuation list and removes the first present mark, nopr times. The specific marks removed from a sentence are therefore arbitrary, and this was done without a fixed seed — the corpus is a single fixed realisation.
• —Sentences originate from BanglaParaphrase, so the domain is that corpus's — largely formal/news-style written Bangla. Expect degradation on conversational or ASR-transcript text, which is the usual real-world target for punctuation restoration.
• —This is not the paper's exact split — see Relationship to the published Jatikarok results.

Citation

@inproceedings{bijoy2023advancing,
  title={Advancing bangla punctuation restoration by a monolingual transformer-based method and a large-scale corpus},
  author={Bijoy, Mehedi Hasan and Faria, Mir Fatema Afroz and Sobhani, Mahbub E and Ferdoush, Tanzid and Shatabda, Swakkhar},
  booktitle={Proceedings of the First Workshop on Bangla Language Processing (BLP-2023)},
  pages={18--25},
  year={2023}
}

License

MIT, matching the Jatikarok reference implementation.