datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
titulm-bangla-corpus
TituLM Bangla Corpus
This dataset is associated with the paper TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking
TituLM Bangla Corpus is one of the largest Bangla clean corpus prepared for pretraining, continual pretraining or fine-tuning Large Language Model(LLM) for improving Bangla text generation capability.
This dataset contains diverse sources and categories of Bangla text. The largest part of this dataset contains filtered common crawled datasets. As we saw… See the full description on the dataset page: https://huggingface.co/datasets/hishab/titulm-bangla-corpus.titulm-bangla-mmlu
Titulm Bangla MMLU
Read the paper for details: https://arxiv.org/abs/2502.11187
Citation
@misc{nahin2025titullmsfamilybanglallms,
title={TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking},
author={Shahriar Kabir Nahin and Rabindra Nath Nandi and Sagor Sarker and Quazi Sarwar Muhtaseem and Md Kowsher and Apu Chandraw Shill and Md Ibrahim and Mehadi Hasan Menon and Tareq Al Muntasir and Firoj Alam},
year={2025},
eprint={2502.11187}… See the full description on the dataset page: https://huggingface.co/datasets/hishab/titulm-bangla-mmlu.bangla-mmlu
Data Summary
We curated multiple-choice questions from various open-source educational websites and textbooks, inspired by the original MMLU dataset (Hendrycks et al., 2020). The dataset includes multiple-choice questions from different Bangladeshi exams, such as job exams, the Bangladesh Civil Service Exam, and undergraduate admission exams. In Figure 7, we report category wise distributions.
Bangla MMLU Dataset Overview
The Bangla MMLU dataset consists of a total of 116,503… See the full description on the dataset page: https://huggingface.co/datasets/hishab/bangla-mmlu.commonsenseqa-bn
Dataset Summary
This is the Bangla translated version of the CommonsenseQA dataset. The dataset was translated using a new method called Expressive Semantic Translation (EST). This method combines both Google Machine Translation and LLM-based rewriting of the translation to enhance the expressiveness and semantic accuracy of the translated content.
Dataset Structure
Data instances
Defaults
An example of a 'train' looks as follows:
{… See the full description on the dataset page: https://huggingface.co/datasets/hishab/commonsenseqa-bn.openbookqa-bn
Data Summary
This is the Bangla-translated version of the OpenBookQA dataset. The dataset was translated using a new method called Expressive Semantic Translation (EST), which combines Google Machine Translation with LLM-based rewriting modifications. This method enhances the semantic accuracy and expressiveness of the translated content. OpenBookQA focuses on advanced question-answering, requiring multi-step reasoning, additional common and commonsense knowledge, and rich text… See the full description on the dataset page: https://huggingface.co/datasets/hishab/openbookqa-bn.piqa-bn
Dataset Summary
This is the translated version of the PIQA LLM evaluation dataset. The dataset was translated using a new method called Expressive Semantic Translation (EST), which combines Google Translation with LLM-based rewriting. PIQA introduces the task of physical commonsense reasoning and provides a corresponding benchmark for understanding physical interactions in everyday situations. It focuses on atypical solutions to practical problems, inspired by instructional guides… See the full description on the dataset page: https://huggingface.co/datasets/hishab/piqa-bn.boolq_bn
Dataset Summary
BoolQ Bangla (BN) is a question-answering dataset for yes/no questions, generated using GPT-4. The dataset contains 15,942 examples, with each entry consisting of a triplet: (question, passage, answer). The questions are naturally occurring, generated from unprompted and unconstrained settings. Input passages were sourced from Bangla Wikipedia, Banglapedia, and News Articles, and GPT-4 was used to generate corresponding yes/no questions with answers.
The dataset was… See the full description on the dataset page: https://huggingface.co/datasets/hishab/boolq_bn.MegaBNSpeech_Test_Data
MegaBNSpeech Test Data
To evaluate the performance of the models, we used four test sets. Two of these were developed as part of the MegaBNSpeech corpus, while the remaining two (Fleurs and Common Voice) are commonly used test sets that are widely recognized by the speech community.
Use dataset library:
from datasets import load_dataset
dataset = load_dataset("hishab/MegaBNSpeech_Test_Data")
Reported Word error rate (WER) /character error rate (CER) on four test… See the full description on the dataset page: https://huggingface.co/datasets/hishab/MegaBNSpeech_Test_Data.pia-torchtune-bn-singleBanglaLLM-alpaca-mergedhishab-pr-bn-v1
Hishab PR Bengali v1
Dataset Description
This dataset contains Bengali text punctuation restoration data in a conversational format. The dataset is designed for training and evaluating models to restore punctuation in Bengali text. Each conversation consists of a human providing unpunctuated Bengali text and a GPT assistant providing the same text with proper punctuation restored.
Dataset Summary
Language: Bengali (bn)
Task: Punctuation Restoration… See the full description on the dataset page: https://huggingface.co/datasets/hishab/hishab-pr-bn-v1.MushanWGLOBEEvalhishab-synthetic-v2