Datasetpublic

istiaqfuad/bangla-english-banglish-pairs

Bangla-English-Banglish Trilingual Pairs Overview This dataset provides contrastive training pairs for fine-tuning trilingual (Bangla / Banglish / English) sentence embedding models (such as BGE-M3). It is designed to impart robustness to Banglish spelling variation. The dataset is combined from two main sources: LLM-generated Banglish spelling variants. The OPUS-100 EN-BN parallel corpus. Included Files File Rows Size Description… See the full description on the dataset page: https://huggingface.co/datasets/istiaqfuad/bangla-english-banglish-pairs.

sourceHugging Facecc-by-4.0updated 4mo agoView on Hugging Face

0likes52downloads

Dataset Card

Bangla-English-Banglish Trilingual Pairs

Overview

This dataset provides contrastive training pairs for fine-tuning trilingual (Bangla / Banglish / English) sentence embedding models (such as BGE-M3). It is designed to impart robustness to Banglish spelling variation.

The dataset is combined from two main sources:

1.LLM-generated Banglish spelling variants.
2.The OPUS-100 EN-BN parallel corpus.

Included Files

File	Rows	Size	Description
`training_pairs.jsonl`	2,405,946	~1.04 GB	Merged training set containing both Banglish and EN↔BN pairs with `source` tags.
`en_bn_pairs_filtered.jsonl`	1,815,832	~797 MB	Cleaned EN↔BN pairs generated from OPUS-100.
`banglish_pairs_filtered.jsonl`	590,114	~179 MB	Cleaned Banglish pairs (spelling variants generated via LLM).

1. Merged Training Set (`training_pairs.jsonl`)

The main training file combines both sources. Due to class imbalance, it includes a source tag ("banglish" or "en_bn") which is useful for weighted random sampling during training to prevent cross-lingual pairs from drowning out the spelling variant pairs.

Schema

json

{
  "anchor": "...",
  "positive": "...",
  "negative": "...",
  "source": "banglish" | "en_bn"
}

2. EN↔BN Pairs (`en_bn_pairs_filtered.jsonl`)

Generated from the Helsinki-NLP/opus-100 bn-en train split. Each parallel pair generates two triplets: EN→BN and BN→EN. Single-word pairs and degenerate pairs have been filtered out.

3. Banglish Pairs (`banglish_pairs_filtered.jsonl`)

Generated from the OPUS-100 bn-en train split using an LLM to produce Banglish spelling variants per sentence (e.g., standard, alternate, abbreviated).

Structure:

—anchor: English, Bangla, or Banglish sentence.
—positive: Banglish variant (meaning-preserving spelling variation).
—negative: Random sentence from another pair (for contrastive learning).

The OPUS-100 dataset portion uses data from the Helsinki-NLP/opus-100 corpus.