sello-ralethe/SA-Parallel-Corpora

SA-Parallel-Corpora

Sentence-aligned English to isiZulu, isiXhosa, Sesotho and Sepedi bitext, drawn from South African government publications.

Produced for the doctoral thesis Injecting Commonsense Knowledge into Pretrained Language Models for Low Resource Languages (University of Cape Town, 2026). Code at https://github.com/sello-ralethe/SA-knowledge

Structure

One configuration per language pair, each with train, validation and test splits. Splits are assigned at document level, so no source document contributes sentences to more than one split.

Each row carries source_en, target, a LaBSE similarity score, a confidence label (high above 0.85, medium from 0.65 to 0.85), word counts for both sides, and doc_id for provenance.

Loading

from datasets import load_dataset

pairs = load_dataset("sello-ralethe/SA-Parallel-Corpora", "eng-zul")
test  = load_dataset("sello-ralethe/SA-Parallel-Corpora", "eng-zul", split="test")

Method

Documents were matched by language-indexed URL stem. Alignment is two pass: Gale and Church length-based dynamic programming with a length ratio estimated per pair, then semantic refinement by beam search over LaBSE cosine similarity. Output is restricted to one-to-one links.

The confidence threshold is not equally strict across languages, because the scoring model covers isiZulu, isiXhosa and Sesotho but not Northern Sotho. The raw score is retained so tiers can be re-cut per language.

Licence

CC BY 4.0, subject to the terms of the originating government departments.