uctnlp/mzansi-text

MzansiText MzansiText is a curated multilingual pretraining corpus for all eleven official South African languages. Dataset details Splits: 3,943,584 train rows, 19,379 validation rows, and 19,341 test rows lang values: afr, eng, nbl, nso, sot, ssw, tsn, tso, ven, xho, zul Schema: { "text": "string", "lang": "string" } Validation and test sets are capped at approximately 2M tokens per language to prevent high-resource languages from dominating early… See the full description on the dataset page: https://huggingface.co/datasets/uctnlp/mzansi-text.