VelkroLM/african-languages-filtered

Filtered African-language datasets

This repository contains non-destructive filtered derivatives of publicly accessible Hugging Face datasets relevant to Hausa, Nigerian languages, and selected African languages. The original repositories remain the authoritative sources and were not modified.

Scope and provenance

Each JSONL file preserves the source repository, source split, and source row index in _source_repo, _source_split, and _source_row_index. Per-repository counts and SHA-256 checksums are in `manifest.json`, with individual audit files under `audit/`.

The first publication wave contains 11 audited derivatives and 416,020 retained rows (397.3 MiB of JSONL). The candidate discovery and deferred-source catalog is maintained separately in the accompanying audit report.

Filtering rules

The transformation normalizes Unicode to NFC and collapses whitespace; removes null or empty text; removes exact duplicate normalized records; removes URLs, email addresses, bot/prompt-injection artifacts, and obvious social-spam patterns; removes only an allowlisted set of parenthetical English glosses; and removes English-heavy monolingual records using a conservative threshold of at least three unambiguous English prose words and an English-word ratio of at least 0.20. Bilingual and parallel corpora are not filtered by deleting their English side, because that would destroy alignment and translation utility. ASR/audio sources are catalogued separately unless a text-only derivative was safely produced.

License and responsible use

Every derivative must be used subject to the original source repository’s license, attribution requirements, dataset-card terms, and any additional restrictions. This repository does not change upstream copyrights or grant a blanket license. In particular, some corpora contain copyrighted source material or terms-of-use requirements even when the compilation carries a permissive license. Users must inspect the upstream card before redistribution or commercial use.

Validation

In addition to programmatic checks, independent random samples were inspected from the Hausa, Kanuri, and Nupe source datasets. The audit report records those checks and explains why bilingual and audio corpora use modality-aware rules.

Citation

When using a file, cite both this derivative repository and the corresponding upstream repository named in its _source_repo field. See `manifest.json` for the complete mapping.