marsh-mellow/hindi_wikipedia

Hindi Wikipedia Corpus

Dataset Description

The Hindi Wikipedia Corpus is a pure Hindi text dataset derived from the Hindi-language Wikipedia (as.wikipedia.org). It contains cleaned plain text extracted from Wikipedia articles, stripped of all formatting, with non-Hindi characters completely removed. This dataset is designed for language modeling, NLP research, creating Hindi specific tokenizers, and other Hindi-language processing tasks.

Data Processing

1. 1.Crawling
2. 2.Used the mwclient Python library to interact with Wikipedia’s API.
3. 3.Only collected content from the main namespace (namespace=0).
4. 4.Pages that were redirects were automatically resolved to their target articles.
5. 5.Page titles were tracked to avoid processing the same article multiple times.
6. 6.Used mwparserfromhell to strip all MediaWiki formatting (links, templates, etc.).

1. 1.Filtering
2. 2.Applied a Unicode regex to keep only Hindi characters (U+0980–U+097F), Hindi digits, whitespace, and essential punctuation.
3. 3.Excluded pages where more than 90% of the content was punctuation or whitespace after cleaning.
4. 4.Removed very short pages that contained too little meaningful text.

Dataset Format

The dataset is provided as a .jsonl (JSON Lines) file, with each line containing:

{"text": "यह एक प्रकार का अल्कोहल है। ..."}

Example Usage

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("marsh-mellow/hindi_wikipedia")
# Access the first example
print(dataset["train"][0]["text"])

License

The text in this dataset originates from Hindi Wikipedia, which is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0) License.