Datasetpublic

maanka2/somali-web-corpus

SOMALI-WEB-CORPUS V1 This dataset consists of clean, structured, and filtered Somali language text compiled from various online sources. It is designed for training and fine-tuning Somali language models (LLMs) and supporting natural language processing (NLP) research for the Somali language. Dataset Details Language: Somali (so) Format: JSON lines (.jsonl) Data Structure: Each record has a single text field containing a cleaned paragraph. Sources… See the full description on the dataset page: https://huggingface.co/datasets/maanka2/somali-web-corpus.

sourceHugging Facemitupdated 4mo agoView on Hugging Face

1likes25downloads

Dataset Card

SOMALI-WEB-CORPUS V1

This dataset consists of clean, structured, and filtered Somali language text compiled from various online sources. It is designed for training and fine-tuning Somali language models (LLMs) and supporting natural language processing (NLP) research for the Somali language.

Dataset Details

—Language: Somali (so)
—Format: JSON lines (.jsonl)
—Data Structure: Each record has a single text field containing a cleaned paragraph.

Sources

The text is extracted from the following websites:

—BBC Somali
—Shabeele media
—SONNA (Somali National News Agency)
—and more

Cleaning and Filtering

—Deduplication: Repeated paragraphs are discarded to ensure data uniqueness.
—Normalization: Web URLs, email addresses, and invalid top-level domains are removed.