thedeba/bnnews

Bengali News Corpus (2023–2026)

Dataset Summary

The Bengali News Corpus (2023–2026) is a large-scale, high-quality monolingual Bengali dataset consisting of 295,020 cleaned news articles scraped from Prothom Alo (https://www.prothomalo.com), Bangladesh's largest Bengali-language daily newspaper.

The dataset spans over 3.5 years of comprehensive news reporting (January 2023 to August 2026) across various domains including National News, Politics, World News, Business & Economy, Technology, Sports, Entertainment, and Editorial Opinions.

This dataset has been systematically cleaned, normalized, deduplicated, and split into standard train (90%), validation (5%), and test (5%) sets to serve as a benchmark dataset for Bengali Natural Language Processing (NLP), Large Language Model (LLM) pre-training/fine-tuning, text classification, and computational linguistics research.

Dataset Structure

Data Splits

Data Fields

Each record in the dataset consists of the following attributes:

• —`title` (string): The headline of the news article, normalized and stripped of linebreaks.
• —`category` (string): The section/category of the news article (e.g., বাংলাদেশ, বিশ্ব, খেলা, বিনোদন, অর্থনীতি, প্রযুক্তি, মতামত).
• —`content` (string): The full cleaned body text of the article.

Data Instance Example

{
  "title": "প্রেসিডেন্ট হিসেবে শপথ নিয়ে দুর্নীতি বন্ধের ঘোষণা লুলার",
  "category": "বিশ্ব",
  "content": "ব্রাজিলের প্রেসিডেন্ট হিসেবে শপথ নিয়েছেন লুলা দা সিলভা। এর মধ্য দিয়ে রোববার তৃতীয়বারের মতো দেশটির রাষ্ট্রপ্রধানের চেয়ারে বসলেন তিনি। শপথ গ্রহণের পর লুলার প্রতিশ্রুতি, তিনি ব্রাজিলকে ‘নতুন করে গড়ে তুলবেন’, লড়বেন পরিবেশ আর দরিদ্রদের জন্য...\n\nঅনুষ্ঠানে অন্যান্যের মধ্যে উপস্থিত ছিলেন ভাইস প্রেসিডেন্ট জেরাল্ডো আল্কমিন..."
}

Data Collection & Cleaning Methodology

1. Data Collection Architecture

• —Source: Daily XML sitemaps (https://www.prothomalo.com/sitemap/sitemap-daily-YYYY-MM-DD.xml) covering all days from January 1, 2023 to August 24, 2026.
• —Extraction: High-speed multi-threaded scraper (scraper.py) with real-time JSONL output streaming.
• —Coverage: 296,952 total raw articles retrieved.

2. Preprocessing & Quality Pipeline

• —URL & Content Deduplication:
• —Normalized URLs (stripping query parameters and fragments).
• —Computed MD5 content hashes to eliminate 1,900 duplicate re-published articles across sections.
• —Title Normalization:
• —Cleaned embedded linebreaks (\n, \r) from 3,842 titles, ensuring strict single-line headlines.
• —Collapsed multiple consecutive spaces into single spaces.
• —Unicode & Character Cleaning:
• —Stripped non-breaking spaces (\xa0), byte-order marks (\ufeff), and zero-width spaces (\u200b) from 25,169 records while preserving legitimate Bengali Unicode typography (such as ZWNJ \u200c and ZWJ \u200d).
• —HTML & Entity Stripping:
• —Unescaped HTML entities ( , &, ", ', etc.).
• —Stripped embedded HTML markup (<p>, <a>, <span>) using regex sanitization.
• —Quality Filtering:
• —Filtered out 32 ultra-short or empty articles with body text length < 30 characters.

Research Applications & Benchmark Tasks

1. 1.Bengali LLM Pre-training & Continuous Pre-training:
2. 2.High-quality, clean Bengali corpus for training or domain-adapting causal LLMs (e.g., LLaMA-3, Mistral, Gemma, BanglaBERT) on contemporary Bengali text.
3. 3.News Article Category Classification:
4. 4.Multi-class text classification evaluating model accuracy on predicting categories across 120+ distinct news sections.
5. 5.Headline Generation & Text Summarization:
6. 6.Paired content -> title generation tasks for abstractive text summarization in low-resource settings.
7. 7.Media Analysis & Computational Social Science:
8. 8.Longitude analysis of public discourse, economic trends, political reporting, and social dynamics in Bangladesh (2023–2026).

Usage Instructions

Loading with Hugging Face datasets

from datasets import load_dataset

# Load dataset directly from Hugging Face Hub
dataset = load_dataset("YOUR_USERNAME/prothom-alo-bengali-news")

# Inspect splits
print(dataset)
# Output:
# DatasetDict({
#     train: Dataset({ features: ['title', 'category', 'content'], num_rows: 265517 }),
#     validation: Dataset({ features: ['title', 'category', 'content'], num_rows: 14752 }),
#     test: Dataset({ features: ['title', 'category', 'content'], num_rows: 14751 })
# })

# Print a sample from train set
print(dataset["train"][0])

Loading Local Parquet Files

from datasets import load_dataset

dataset = load_dataset("parquet", data_files={
    "train": "data/hf_parquet/train.parquet",
    "validation": "data/hf_parquet/validation.parquet",
    "test": "data/hf_parquet/test.parquet"
})

Licensing & Ethics Statement

• —License: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0).
• —Intended Use: Strictly for non-commercial academic research, educational purposes, and scientific evaluation.
• —Source Copyright: Original article contents belong to Prothom Alo (Mediasaracens Ltd.).
• —Privacy: Publicly available news reporting containing no personal identifiable data beyond public figures mentioned in news events.

Citation & Academic Reference

If you use this dataset or dataset curation methodology in your research paper, please cite it using the following BibTeX entry:

@dataset{bengali_news_corpus_2026,
  author       = {Debashish Roy and Research Team},
  title        = {Bengali News Corpus (2023–2026): A Large-Scale Monolingual Dataset for Bengali NLP},
  year         = {2026},
  publisher    = {Hugging Face},
  journal      = {Hugging Face Datasets Repository},
  howpublished = {\url{https://huggingface.co/datasets/thedeba/bnnews}},
  note         = {Dataset containing 295,020 cleaned Bengali news articles across 2023--2026}
}