rufatronics/african-languages-hplt-filtered

VelkroLM African Languages Corpus

This repository contains a filtered, provenance-preserving text corpus derived from the language-specific HPLT v3.0 shards discovered at hplt-project.org/datasets/v3.0. It is organized by language and source shard so researchers can load only the languages they need. The upstream HPLT project describes its v3.0 data as multilingual web-corpus material; the upstream release, source metadata, and terms remain authoritative.

This publication is deliberately honest about scale. The completed HPLT subset is 3,382,250 retained records and 3.59 GiB of compressed JSONL, not 500 GB. The 500 GB aspiration can only be approached by adding additional legally reusable sources such as larger language shards, parallel corpora, and speech/audio collections after separate license and quality review. The corpus is not inflated with duplicate mirrors or synthetic copies.

Language coverage

File layout

Each language directory contains compressed JSONL output and a matching audit file for every processed source shard.

Hausa/10_1.jsonl.zst
Hausa/10_1.audit.json
Yoruba/8_1.jsonl.zst
...
aggregate_stats.json
manifest.json
README.md

Each retained JSON object contains record_id, language, language_code, script, text, source_dataset, source_revision, source_url, source_record_id, license, license_tier, provenance, quality, and processing_version.

Filtering policy

The pipeline used one active download worker and streamed each Zstandard shard without loading the corpus into memory. It removed empty or malformed rows, obvious credential or spam artifacts, excessive URL spam, low-confidence language rows when a confidence field was provided, and records that violated the minimum quality policy. It did not delete embedded English names or code-switching from otherwise valid African-language text. Parallel corpora and bilingual records must be handled as aligned data and are not made monolingual by deleting one side.

The filter is conservative rather than a claim of perfect language identification. Every shard includes a 200-record seeded reservoir sample in its audit JSON. Random samples should be inspected by downstream users before training, especially because web corpora may contain copyrighted material, duplicated articles, named entities, translated pages, and domain-specific text.

Provenance and licensing

All records in this repository identify the HPLT v3.0 source family and direct source shard URL. The HPLT source page and its accompanying metadata must be cited together with this derivative. The repository metadata uses CC0 for this HPLT-derived subset based on the HPLT release terms; this does not erase rights or restrictions that may attach to the underlying web documents. Users are responsible for complying with applicable law, upstream notices, privacy obligations, and the terms of any source material.

The complete processing manifest is in `manifest.json`, aggregate counts are in `aggregate_stats.json`, and each *.audit.json file records source URL, counts, drop reasons, random sample, output checksum, and processing version.

Loading example

from datasets import load_dataset

ds = load_dataset(
    "VelkroLM/african-languages-corpus",
    data_files={"train": "Hausa/*.jsonl.zst"},
    streaming=True,
)
for row in ds["train"].take(3):
    print(row["text"])

To load another language, replace Hausa with Yoruba, Igbo, Amharic, Somali, or another directory listed above. The compressed JSONL files are intentionally shardable for streaming.

Reproducibility

• —Processing version: 0.2.0.
• —Audit manifest: manifest.json and the local append-only source manifest.
• —Random validation: 200 seeded reservoir samples per shard where the shard contained enough records.
• —Download policy: resumable HTTP Range requests, one active source connection, checksum verification, and deletion of local raw staging only after successful output verification.
• —Publication policy: source originals are not overwritten; filtered derivatives are separate repositories or directories.

Citation

Please cite both the derivative repository and the HPLT project. For a specific record, retain its source_url, source_dataset, and source_record_id fields.

References

1. 1.HPLT v3.0 datasets and download documentation
2. 2.HPLT project
3. 3.VelkroLM African-language corpus processing architecture

Build date: 2026-08-22