tensorlabco/bn_en_banglish_100k_finetune

bnenbanglish100kfinetune

A curated, category-balanced, deduplicated 100,000-row subset of `tensorlabco/bn_en_banglish_v2` (1,370,553 source rows), built for Bangla↔English translation fine-tuning. This is the full, 3-text-field variant; see the companion `tensorlabco/bn_en_100k_finetune` for a bn_text/en_text-only projection of the exact same 100,000 rows and split.

Columns

Construction

1. 1.Word-count filters. story rows keep bn_text word count ≥ 100 (no upper bound — story is naturally clustered around 500 words). news rows are filtered to bn_text word count in [300, 600]; any news category with fewer than 30 rows in that window is dropped entirely.
2. 2.Pool-level deduplication and quality filtering, before any sampling decision. The filtered pool is deduplicated (exact-match, then MinHash/LSH near-duplicate collapse at Jaccard ≥ 0.9) and rows with a banglish_text length wildly disproportionate to bn_text are dropped — both done at the pool level so every category's available supply is a true count of unique, usable content before quotas are decided, and the published banglish_text column is never null.
3. 3.Balanced category quotas via water-filling. story (book) takes its full cleaned supply; the remaining budget is distributed across the 19 news categories, smallest-supply categories getting their full available count and the rest capped at a shared ceiling so the total lands exactly on 100,000.
4. 4.Stratified 80/10/10 split, independently per category, so the category mix is preserved in every split.

Verified on the final selection: 0 near-duplicate `bn_text` pairs (both within-split and cross-split), 0 train/validation/test leakage, 0 `banglish_text` outliers, 0 empty fields.

Dataset structure

100,000 rows total: 80,000 train / 10,000 validation / 10,000 test, across 20 categories.

Known limitations

• —book (28,181 rows) is short of the news categories' shared ceiling and short of its own pre-cleaning pool (29,328) — this is real: the source has ~1,147 duplicate or banglish-outlier story rows and no further unique/usable story content exists in the source to replace them with. The 8 largest news categories absorbed the corresponding extra budget instead.
• —The smallest retained categories (durporobash: 33, anniversary: 94) have single-digit validation/test counts — kept anyway to preserve every sub-category with meaningful supply rather than dropping thin ones for cleaner statistics.

Source

Derived from `tensorlabco/bn_en_banglish_v2` (all 3 original splits combined and treated as one pool before re-filtering and re-splitting).