W4ashabii/root_affix_dictionary

Nepali Root-Affix Dictionary

A word-level morphological segmentation dataset for Nepali, mapping surface words to their root and affix components (e.g. अँकाइनु → अँका + इनु).

Dataset Details

Dataset Description

This dataset was built by scraping the 10th edition of the Nepali dictionary, parsing the scraped entries into structured word/root pairs, and then applying a heuristic rule engine to extend root-affix coverage to surface forms not explicitly listed as dictionary headwords (e.g. inflected or derived forms of a root that is in the dictionary). Each row records the raw surface word, its root, the affix (if any), the +-delimited segmented string, and which process produced that row.

• —Language: Nepali (ne)
• —Curated by: w4ashabii
• —License: MIT

Dataset Sources

• —Source text: Nepali dictionary, 10th edition (https://archive.org/details/nepali-brihat-sabdakosh-nepali-dictionary/page/n11/mode/2up)

Dataset Structure

53,119 rows, single train split, CSV format with 5 columns:

Source values:

• —`dictionary` — root/affix pair taken directly from a dictionary entry
• —`heuristic` — root/affix inferred by the rule engine from a related dictionary entry, not itself a dictionary headword
• —`none` — the word has no affix; Root == Word == Segmented

Dataset Creation

Curation Rationale

Built to support morpheme-boundary prediction for Nepali, particularly generalization to out-of-vocabulary (OOV) roots not seen during training — see the companion fine-tuned model, `w4ashabii/Byt5_OOV_finetuned`.

Source Data

Data Collection and Processing

1. 1.The 10th edition of the Nepali dictionary was scraped for headwords and their listed inflected/derived forms.(https://archive.org/details/nepali-brihat-sabdakosh-nepali-dictionary/page/n11/mode/2up)
2. 2.The scraped entries were parsed into (Word, Root) pairs.
3. 3.A heuristic rule engine extended coverage beyond literal dictionary entries — inferring root/affix splits for related surface forms sharing a dictionary-attested root (e.g. other inflections of a verb root that itself appears in the dictionary).
4. 4.Each resulting row was tagged with its Source (dictionary, heuristic, or none for unsegmented words) to keep the two processes distinguishable downstream.

Who are the source data producers?

Nepali dictionary (10th edition) editors/lexicographers, via automated scraping and heuristic parsing by the dataset curator.

Recommendations

If training a model for OOV generalization on this data, split by Root (not by Word) to avoid leaking the same root across train/val/test via different inflections of it.

Citation

If you use this dataset, please cite the Nepali dictionary, Sabdakosh 2075 10th edition, as the underlying source, along with this repository.

BibTeX:

@misc{root_affix_dictionary,
  author = {w4ashabii},
  title = {Nepali Root-Affix Dictionary},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/w4ashabii/root_affix_dictionary}}
}