Zyroxx66/somali-master-pretraining-corpus

🇸🇴 Somali Master Pretraining Corpus (176.5k Rows)

The Somali Master Pretraining Corpus is a curated, balanced dataset designed for Continued Pre-Training (CPT) and foundational pre-training of Large Language Models (LLMs) in the Somali language (Af-Soomaali).

It addresses the fundamental challenges of low-resource NLP for Somali by combining quality-filtered web knowledge, structured modern domain knowledge, and synthetic narrative intelligence (TinyStories).

🎯 Motivation & Why This Dataset Was Created

Training or adapting LLMs for Somali typically encounters three critical pitfalls:

1. 1.The Political/News Skew of Web Data: Scraped web data is heavily dominated by political news, conflict reports, and dry government press releases. Models trained solely on web data sound like formal news anchors and struggle with natural conversation, creative storytelling, and emotional reasoning.
2. 2.Lexical Hallucinations in Translations: Machine-translating English datasets into Somali causes severe vocabulary hallucinations (e.g., inventing pseudo-Somali words for non-native fauna or modern concepts).
3. 3.The Syntactic Foundation Gap: Small language models (<3B parameters) require high-density, simple grammatical examples to master Subject-Verb-Object (SVO/SOV) agreement, noun-gender particles (wuxuu vs. waxay), and logical cause-and-effect.

The Solution: This corpus resolves these issues through a 57/24/19 balanced mixture that combines formal linguistic breadth, narrative fluency, and modern technical/scientific vocabulary.

📊 Dataset Composition & Mixture

The dataset contains 176,498 deduplicated, globally shuffled rows across three complementary pillars:

🔬 Methodology & Synthetic Generation

1. Gemma 4 MoE TinyStories Engine (42,000 Rows)

Following the principles of Microsoft Research’s TinyStories paper (Eldan & Li, 2023), 42k child-level short stories were generated using Google's Gemma 4 26B-A4B MoE in 4-bit dynamic quantization (UD-IQ4_XS).

• —Combinatorial Matrix: Prompts were deterministically generated across 30 Character Archetypes, 20 Cultural Settings, 25 Plot Conflicts, and 10 Moral Lessons ($30 \times 20 \times 25 \times 10 = 150,000$ non-overlapping permutations).
• —Anti-Overfitting Prompting: Hardcoded character names were explicitly forbidden; the model dynamically selected culturally authentic Somali names (Liibaan, Warsame, Sahra, Guuleed, Beydan, Canab, Hibaaq, etc.) per story.
• —Distributed Generation: Computed in parallel across dual Nvidia Tesla T4 GPU clusters using deterministic mathematical sharding.

2. High-Quality Web Sampling (100,000 Rows)

• —Sourced from somaliweb-v1, filtered for rows with quality_score >= 0.93.
• —Balanced across sources (hplt2-so, cc100-so, wikipedia-so) to prevent monolithic source bias.

3. Domain CPT Deep Dives (34,498 Rows)

• —Monolingual Somali structured articles covering topics traditionally underrepresented in Somali NLP: Mental health (fayoqabka maskaxda), Renewable Energy (tamarta la cusboonaysiin karo), Digital Finance, and Science.

🛠️ Dataset Format & Usage

The dataset is partitioned into clean, optimized Apache Parquet shards. Every row contains two standardized fields:

{
  "text": "Maalin maalmaha ka mid ah, wiil yar oo la yiraahdo Liibaan ayaa marayay waddo aad u mashquul badan...",
  "source": "tinystories_gemma4"
}

Loading with Hugging Face datasets:

from datasets import load_dataset

# Stream or download the full shuffled dataset
dataset = load_dataset("Zyroxx66/somali-master-pretraining-corpus", split="train")

print(f"Total Rows: {len(dataset):,}")
print(dataset[0])

Continued Pre-Training with Unsloth / PyTorch:

from transformers import TrainingArguments
from trl import SFTTrainer

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    dataset_num_proc=4,
    packing=True, # Recommended for high training throughput
    args=TrainingArguments(
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        warmup_ratio=0.05,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=10,
        output_dir="outputs",
    ),
)
trainer.train()

📜 License

This dataset is distributed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). You are free to use, modify, distribute, and integrate this corpus into commercial and research language models.

🙏 Acknowledgements & References

• —Khaled Yusuf for curating and releasing `somaliweb-v1`.
• —Ronen Eldan & Yuanzhi Li (Microsoft Research) for the TinyStories methodology.
• —Google DeepMind & Unsloth for the Gemma 4 Mixture-of-Experts architecture and quantization tooling.