Datasetpublic

ananddey/assamese-wiki-corpus

Assamese Wiki Corpus (ananddey/assamese-wiki-corpus) A clean, large scale Assamese text corpus spanning wiki articles, literary works, dictionary entries, and quotations, curated for language model pre training, fine tuning, and NLP research. Total characters: 65,899,040Approximate tokens : 16,474,760 (16.5M) Fields Field Type Description id int64 Wikimedia page ID title string Page title text string Cleaned plain text content source class_label… See the full description on the dataset page: https://huggingface.co/datasets/ananddey/assamese-wiki-corpus.

sourceHugging Facecc-by-sa-4.0updated 2mo agoView on Hugging Face

1likes27downloads

Dataset Card

Assamese Wiki Corpus (ananddey/assamese-wiki-corpus)

A clean, large scale Assamese text corpus spanning wiki articles, literary works, dictionary entries, and quotations, curated for language model pre training, fine tuning, and NLP research.

Total characters: 65,899,040 Approximate tokens : 16,474,760 (16.5M)

Fields

Field	Type	Description
`id`	int64	Wikimedia page ID
`title`	string	Page title
`text`	string	Cleaned plain text content
`source`	class_label	Source project (wikipedia, wikisource, wiktionary, wikiquote)
`url`	string	URL to the original page

Splits

Split	Pages	Chars	~Tokens
train	24,829	63,891,401	15,972,850
validation	800	2,007,639	501,909

Usage

python

from datasets import load_dataset

dataset = load_dataset("ananddey/assamese-wiki-corpus")
print(dataset["train"][0])

For language model training

This dataset provides clean Assamese text ready for tokenization. Strip the metadata columns:

python

dataset = dataset.remove_columns(['id', 'title', 'source', 'url'])

License

Content from this projects is available under CC BY-SA 4.0.

Citation

bibtex

@dataset{ananddey_assamese_wiki_corpus,
  author    = {Anand Dey},
  title     = {Assamese Wiki Corpus},
  year      = {2026},
  url       = {https://huggingface.co/datasets/ananddey/assamese-wiki-corpus}
}