Datasetpublic

premmm/nepali-bench

NepaliBench 🏔️ A rigorous evaluation benchmark for Nepali language models. Why this exists There is no standard, publicly reproducible benchmark for evaluating Nepali LLMs. This dataset was created after a systematic evaluation of himalaya-ai's NanochatGPT and Gemma fine-tune revealed that models claiming Nepali capability had no shared evaluation standard to measure against. Dataset 100 carefully curated evaluation examples across 8 categories:… See the full description on the dataset page: https://huggingface.co/datasets/premmm/nepali-bench.

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

1likes37downloads

Dataset Card

NepaliBench 🏔️

A rigorous evaluation benchmark for Nepali language models.

Why this exists

There is no standard, publicly reproducible benchmark for evaluating Nepali LLMs. This dataset was created after a systematic evaluation of himalaya-ai's NanochatGPT and Gemma fine-tune revealed that models claiming Nepali capability had no shared evaluation standard to measure against.

Dataset

100 carefully curated evaluation examples across 8 categories:

Category	Code	Count	Description
Nepali Factual Knowledge	FACT	20	Geography, history, constitution, symbols
Nepali Cultural Knowledge	CULT	15	Festivals, ethnic traditions, proverbs
Nepali Language Competence	LANG	15	Grammar, script, translation
Mathematical Reasoning	MATH	15	Arithmetic in Nepali cultural context (mana/pathi, ropani, bigha)
Logical Reasoning	REASON	10	Syllogisms, analogies, sequences
Instruction Following	INST	10	Format compliance, length constraints
Nepali NLP Tasks	NLP	10	NER, sentiment, summarization, language ID
Safe Behavior	SAFE	5	Appropriate refusal, uncertainty acknowledgment

Schema

json

{
  "id": "NB-FACT-001",
  "category": "Nepali Factual Knowledge",
  "subcategory": "geography",
  "prompt_ne": "सगरमाथाको उचाइ कति छ?",
  "prompt_en": "What is the height of Mount Everest?",
  "reference_answer_ne": "८,८४८.८६ मिटर",
  "reference_answer_en": "8,848.86 meters",
  "difficulty": "easy",
  "answer_type": "exact",
  "rubric": null,
  "verified_source": "Nepal-China Joint Survey 2020",
  "tags": ["geography", "everest", "height"]
}

Answer Types

—exact — model answer must match reference exactly (or contain exact figure)
—contains — model answer must contain key information from reference
—rubric — model answer evaluated against rubric criteria

Design Principles

—All FACT answers verified against official sources (Nepal Government, UN records, Constitution of Nepal 2072)
—Regional diversity: Madhesh/Terai culture explicitly represented (Chhath, Tharu Maghi, Maithili language)
—Nepali unit systems used in MATH (mana/pathi, ropani/aana, bigha/kattha)
—Difficulty genuinely distributed: 41% easy, 40% medium, 19% hard

Origin

Created as part of an independent audit of himalaya-ai's Nepali AI model ecosystem (June 2026). Two PRs were filed against himalayagpt-0.5b and himalayagpt-0.5b-it fixing critical GenerationMixin bugs discovered during evaluation.

Author

Premanand Pathak (premmm) — AI Engineer, Kathmandu, Nepal

Citation

bibtex

@dataset{pathak2026nepalibench,
  author    = {Premanand Pathak},
  title     = {NepaliBench: A Benchmark for Nepali Language Model Evaluation},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/premmm/nepali-bench}
}