smcproject/ml-wiki-sentences

ml-wiki-sentences


Malayalam Wikipedia sentences extracted from article text, segmented into individual sentences.

Dataset Description

This dataset contains sentence-segmented text from Malayalam (ml) Wikipedia articles. Each row represents a single sentence extracted from Wikipedia articles, with metadata linking it back to the source article.

Data Source

• —Source: Malayalam Wikipedia (ml.wikipedia.org)
• —Dump Date: March 2025
• —Original Dump: Wikimedia Enterprise HTML dumps (20250320)
• —Articles Processed: 88,832 articles
• —Total Sentences: 2,250,219 sentences

Language

• —Language: Malayalam (ml)
• —Script: Malayalam (ml)

Dataset Schema

Dataset Creation

1. 1.Downloaded enterprise HTML dumps from Wikimedia
2. 2.Extracted article HTML and converted to Parquet format
3. 3.Used tree-sitter-html (Rust) to extract plain text from HTML
4. 4.Used sentencex (Rust) for sentence segmentation

Source code: https://github.com/santhoshtr/wikisentences

Licensing

Creative Commons Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0)

This dataset is derived from Wikipedia content, which is available under CC BY-SA 3.0. The source text is licensed under the same terms.

See: https://creativecommons.org/licenses/by-sa/3.0/

Citation

If you use this dataset in your research or projects, please cite:

ml-wiki-sentences Dataset (2025). Malayalam Wikipedia sentences.
Available at: https://huggingface.co/datasets/smcproject/ml-wiki-sentences

Contact

For questions or issues, please open an issue on the repository.

Dataset automatically generated from Malayalam Wikipedia dumps.