Datasetpublic

uctnlp/mzansi-text-deduplicated

MzansiText — deduplicated release This is a conservatively deduplicated release of MzansiText, a multilingual pretraining corpus for all eleven official South African languages. Use the original release to reproduce the paper and its trained models. Use this release for new experiments where cross-source duplicate documents should be removed. Dataset details Splits: 3,744,654 train rows, 19,940 validation rows, and 19,784 test rows Total: 3,784,378 rows lang… See the full description on the dataset page: https://huggingface.co/datasets/uctnlp/mzansi-text-deduplicated.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

1likes538downloads

Dataset Card

MzansiText — deduplicated release

This is a conservatively deduplicated release of MzansiText, a multilingual pretraining corpus for all eleven official South African languages.

Use the original release to reproduce the paper and its trained models. Use this release for new experiments where cross-source duplicate documents should be removed.

Dataset details

—Splits: 3,744,654 train rows, 19,940 validation rows, and 19,784 test rows
—Total: 3,784,378 rows
—lang values: afr, eng, nbl, nso, sot, ssw, tsn, tso, ven, xho, zul
—Schema: text (string), lang (string)
—All eleven languages occur in every split.

Validation and test sets are capped at approximately 2M tokens per language to prevent high-resource languages from dominating early stopping.

Deduplication

Deduplication is performed before deterministic splitting. Each document is normalized with Unicode NFC, whitespace is collapsed and trimmed, and exact duplicates are identified by (lang, SHA-256(normalized text)). The language is part of the key so identical text labelled as different languages is retained.

The process removed 197,926 duplicate rows:

Language label	Rows removed
`afr`	165,623
`zul`	19,156
`xho`	12,910
`tso`	231
`sot`	3
`eng`	1
`nbl`	1
`nso`	1
Total	197,926

An audit of the combined train, validation, and test splits found no remaining duplicates under this normalization and per-language key.

Usage

python

from datasets import load_dataset

dataset = load_dataset("uctnlp/mzansi-text-deduplicated")
print(dataset)

Reproducibility

—Original paper-reproduction dataset: uctnlp/mzansi-text
—Deduplication implementation and config: Anri-Lombard/sallm#123
—Paper: arXiv:2603.20732

Citation

bibtex

@misc{lombard2026mzansitextmzansilmopencorpus,
      title={MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages},
      author={Anri Lombard and Simbarashe Mawere and Temi Aina and Ethan Wolff and Sbonelo Gumede and Elan Novick and Francois Meyer and Jan Buys},
      year={2026},
      eprint={2603.20732},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.20732},
}

License

Apache License 2.0