mzansitext
mzansi-text
MzansiText
MzansiText is a curated multilingual pretraining corpus for all eleven official South African languages.
Dataset details
Splits: 3,943,584 train rows, 19,379 validation rows, and 19,341 test rows
lang values: afr, eng, nbl, nso, sot, ssw, tsn, tso, ven, xho, zul
Schema:
{
"text": "string",
"lang": "string"
}
Validation and test sets are capped at approximately 2M tokens per language to prevent high-resource languages from dominating early… See the full description on the dataset page: https://huggingface.co/datasets/uctnlp/mzansi-text.mzansi-text-deduplicated
MzansiText — deduplicated release
This is a conservatively deduplicated release of MzansiText, a multilingual pretraining corpus for all eleven official South African languages.
Use the original release to reproduce the paper and its trained models. Use this release for new experiments where cross-source duplicate documents should be removed.
Dataset details
Splits: 3,744,654 train rows, 19,940 validation rows, and 19,784 test rows
Total: 3,784,378 rows
lang… See the full description on the dataset page: https://huggingface.co/datasets/uctnlp/mzansi-text-deduplicated.mzansi-text-tokenized
MzansiText Tokenized
Document-tokenized MzansiText used by the later SALLM pretraining runs.
Tokenizer: custom 65,536-vocabulary BPE tokenizer
Splits: 3,943,584 train, 19,379 validation, and 19,341 test rows
Each raw document remains one row and is truncated to at most 2,048 tokens
Sequence lengths are variable; this is not the fixed-length packed/chunked artifact used by the original MzansiLM-125M run
The full corpus includes WURA, ParaCrawl, and the isiXhosa/isiZulu mC4… See the full description on the dataset page: https://huggingface.co/datasets/uctnlp/mzansi-text-tokenized.