CoolFace
Datasetpublic

LocaleNLP/AfriCorpus-v1

AfriCorpus v1 AfriCorpus-v1 is the first public release of LocaleNLP's audited, deduplicated, and quality-filtered African language corpus. Built to power the AfriLION LLM project, this dataset directly addresses the Tokenizer Fertility problem that causes all current LLMs to underperform on African languages. Key Statistics Language Code Script CC-100 Source Status Wolof wo Latin CC-100 Audited Swahili sw Latin CC-100 Audited Hausa ha Latin + Ajami… See the full description on the dataset page: https://huggingface.co/datasets/LocaleNLP/AfriCorpus-v1.

sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face
0likes13downloads
Dataset Card

AfriCorpus v1

AfriCorpus-v1 is the first public release of LocaleNLP's audited, deduplicated, and quality-filtered African language corpus. Built to power the AfriLION LLM project, this dataset directly addresses the Tokenizer Fertility problem that causes all current LLMs to underperform on African languages.

Key Statistics

LanguageCodeScriptCC-100 SourceStatus
WolofwoLatinCC-100Audited
SwahiliswLatinCC-100Audited
HausahaLatin + AjamiCC-100Audited
YorubayoLatinCC-100Audited
AmharicamGe'ez (Ethiopic)CC-100Audited
TigrinyatiGe'ez (Ethiopic)CC-100In Progress
SomalisoLatinCC-100In Progress
IgboigLatinCC-100In Progress
ZuluzuLatinCC-100In Progress

Quality Assurance Pipeline

Every document in this corpus has passed through a 7-stage pipeline:

  1. 1.Download — CC-100 .txt.xz source files from StatMT.
  2. 2.Language-ID Filter — langdetect with confidence threshold > 0.90.
  3. 3.Text Cleaning — URL removal, HTML stripping, control character normalization.
  4. 4.Deduplication — MinHash LSH (threshold 0.85, 128 permutations), including cross-lingual dedup.
  5. 5.Length Filter — Only sentences with 20–2048 whitespace tokens are kept.
  6. 6.JSONL Sharding — 100k lines per shard for streaming compatibility.
  7. 7.Upload — Published here with provenance metadata on every record.

Critical Design Decisions

Ge'ez Script Handling

Amharic and Tigrinya use the Ge'ez (Ethiopic) script which has ~500 base syllabic characters. Each combination is a unique glyph, leading to thousands of distinct characters. Training on this corpus requires character_coverage=0.9999 in SentencePiece. Do not lower this value or your tokenizer will produce <0xE1><0x88><0xA0> byte-fallback tokens instead of actual Ge'ez glyphs, silently corrupting Amharic model training.

Equal Upsampling

Wolof has ~40MB of CC-100 data; Swahili has ~6.6GB. A proportionally-weighted tokenizer devotes most of its vocab budget to Swahili, leaving Wolof with ~200 tokens that fragment every word into 5–6 pieces. Our tokenizer training script upsamples Wolof 150x to achieve equal representation.

Lang ID Tokens

Every document is prepended with a language ID token ([WO], [SW], [HA], [AM], etc.) during tokenizer training. This enables the model to condition on language at inference time — critical for code-switching and per-language perplexity measurement.

Usage

python
from datasets import load_dataset

# Load a specific language
ds = load_dataset("LocaleNLP/AfriCorpus-v1", split="wo")
print(ds[0])
# {'text': 'Nanga def, baal ma.', 'lang': 'wo', 'lang_name': 'Wolof', 
#  'token_count': 5, 'source': 'cc100'}

# Load all languages
ds_all = load_dataset("LocaleNLP/AfriCorpus-v1")

Citation

If you use this dataset, please cite:

bibtex
@dataset{africorpus_v1_2026,
  title   = {AfriCorpus v1: Audited African Language Corpus for LLM Training},
  author  = {Jagne, Alieu and LocaleNLP Team},
  year    = {2026},
  url     = {https://huggingface.co/datasets/LocaleNLP/AfriCorpus-v1},
  license = {cc-by-4.0}
}

Related Resources