tonibirat/neBrahma-Nepali-Pretrain-Corpus

neBrahma Nepali Pretrain Corpus P2b

Dataset Summary

The neBrahma Nepali Pretrain Corpus P2b is a large-scale, production-grade Nepali text corpus assembled and certified for language model pretraining. It contains 20,321,968 documents and 1.845 billion tokens of clean, verified Devanagari Nepali text, drawn from four diverse sources and processed through an eight-stage cleaning and quality pipeline.

This corpus serves as the training data for neBrahma-llm - a Nepali multimodal LLM (text + ASR + TTS) built from scratch in C++/CUDA. The corpus is certified FIT TO TRAIN with zero warnings and a 300-document human spot-check confirming linguistic quality.

EDA Visualizations

Pipeline Overview

The corpus was constructed through an eight-stage pipeline:

1. 1.Fetch - stream raw documents from source APIs (HuggingFace datasets, IRIIS, ASR text)
2. 2.Clean - Unicode NFC normalization, Devanagari font conversion (Preeti/Kantipur → Unicode), script-ratio pre-filter
3. 3.Exact dedup - SHA-1 hash deduplication within each source
4. 4.Language ID - GlotLID filtering: keep npi_Deva (Nepali Devanagari) with score >= 0.6; line-level filtering for mixed-language documents
5. 5.Cross-source exact dedup - SHA-1 deduplication across all sources (merged)
6. 6.Near-dedup - MinHash LSH at Jaccard >= 0.8; removes near-duplicate paragraphs
7. 7.Quality annotation + filter - 25+ signal gates: word count, Devanagari ratio, digit ratio, Latin ratio, KenLM perplexity, n-gram repetition, stopword ratio, and more
8. 8.Split + pack - deterministic hash-keyed train/val/test split, then SentencePiece BPE tokenization into packed uint16 shards for training

Source Composition and Keep Rates

Quality Filter Gate Analysis

The chart below shows the top rejection gates (gates overlap - a document may fail multiple).

Document Length Distribution

Distribution of document lengths (in characters) across the quality-filtered corpus (sampled from 200K documents).

Devanagari Ratio Distribution

All retained documents have Devanagari ratio = 1.0 (pure script, no Latin contamination).

Character Category Breakdown

Top-100 Nepali Words by Frequency

Token Zipf Distribution

Vocabulary frequency distribution follows the expected Zipfian curve. 32,000-token BPE vocabulary with 97.16% coverage (31,091 of 32,000 tokens seen in corpus).

Dataset Structure

Data Fields

Example Record

{
  "id": "indiccorp:npi_Deva_2847193",
  "text": "नेपाल सरकारले नयाँ शिक्षा नीति लागू गर्ने घोषणा गरेको छ। यो नीतिले प्राथमिक विद्यालयदेखि विश्वविद्यालय तहसम्मका विद्यार्थीहरूलाई प्रभाव पार्नेछ।",
  "source": "indiccorp"
}

Statistics

Corpus-Level

Per-Source Document Counts

Quality Certification

Verdict   : FIT TO TRAIN
hard_ok   : true
warnings  : [] (zero)
spot_check: 300 documents - all pass
binary_checks: PASS (SHA-256 integrity, shard contract)
reversible: true (kept ∪ rejected == raw input)

Usage

from datasets import load_dataset

# Load the full corpus (~8GB parquet download)
ds = load_dataset("tonibirat/neBrahma-Nepali-Pretrain-Corpus")
print(ds)
# DatasetDict({'train': Dataset({features: ['id', 'text', 'source'], num_rows: 20321968})})

# Filter by source
indiccorp_only = ds["train"].filter(lambda x: x["source"] == "indiccorp")

# Iterate (streaming for large-scale use)
ds_stream = load_dataset("tonibirat/neBrahma-Nepali-Pretrain-Corpus", streaming=True)
for example in ds_stream["train"]:
    text = example["text"]   # Nepali Devanagari text
    src  = example["source"] # "indiccorp" | "fineweb2" | "iriis" | "sagarmatha"
    break

Limitations

• —IRIIS crowd-source noise: The IRIIS subset relies on crowd-sourced transcriptions which may contain latent inaccuracies despite passing the quality filter.
• —No demographic metadata: No speaker, regional dialect, or domain metadata is included. Source provenance is available via the source field.
• —Sagarmatha ASR fraction is small: The Sagarmatha ASR text subset (27,733 docs, 0.1%) consists of short utterance-level transcriptions and has a lower keep rate (54.6%) due to the minimal-sentence nature of ASR transcripts.
• —EDA sample: Document-level EDA visualizations (length distribution, character breakdown, word frequency) were computed on a 200K-document sample, not the full 20.3M. Shard-level token statistics are exact (computed from all 21 shards).
• —Per-source license: IndicCorp v2 is CC0; FineWeb-2 is ODC-By (attribution required); IRIIS is distributed for research use. Verify compliance with your use case.

Citation

If you use this dataset, please cite:

@misc{neBrahma2026,
  title        = {neBrahma-llm: A Nepali Multimodal Language Model Trained from Scratch},
  author       = {Gautam, Toni Birat},
  year         = {2026},
  institution  = {Final Year Project},
  howpublished = {\url{https://github.com/ToniBirat7/neBrahma-llm}},
  note         = {Corpus: neBrahma Nepali Pretrain Corpus P2b --
                  20.3M docs / 1.845B tokens / 4 sources / FIT TO TRAIN}
}