rufatronics/hausa-10phase-synthetic-training-corpus
Hausa 10-Phase Synthetic Training Corpus Summary This repository contains a large, structured, synthetic Hausa-language corpus organized into 10 curriculum phases. The curriculum moves from beginner greetings and everyday services to procedure explanation, intent classification, text transformation, contextual reasoning, domain question answering, structured extraction, evidence-grounded question answering, and safety-oriented robustness tasks. The corpus contains… See the full description on the dataset page: https://huggingface.co/datasets/rufatronics/hausa-10phase-synthetic-training-corpus.
Hausa 10-Phase Synthetic Training Corpus
Summary
This repository contains a large, structured, synthetic Hausa-language corpus organized into 10 curriculum phases. The curriculum moves from beginner greetings and everyday services to procedure explanation, intent classification, text transformation, contextual reasoning, domain question answering, structured extraction, evidence-grounded question answering, and safety-oriented robustness tasks.
The corpus contains 75,693,486 records and 100,000,063,399 validated raw UTF-8-equivalent bytes, distributed across 10 logical batches. Each logical batch contains ten approximately 1 GB raw phase shards, giving a total of 100 phase shards. The raw logical size and the compressed Hugging Face storage size are reported separately because gzip compression makes them substantially different.
Important: This is a synthetic, template/combinatorial corpus. It is not a collection of naturally occurring Hausa text, and its size must not be interpreted as 100 GB of unique linguistic knowledge.
Intended uses
The corpus is intended for Hausa NLP prototyping, data-pipeline testing, curriculum-learning experiments, structured prediction, instruction-format experiments, safety-task prototyping, and baseline model development. It can also be used to test loading, sharding, validation, deduplication, and evaluation infrastructure for a low-resource language.
The corpus should not be used by itself as evidence of fluent Hausa generation, as a substitute for community-reviewed language resources, or as the sole training source for a production model. High-stakes medical, legal, financial, civic, or safety decisions require independently verified sources and qualified human review.
Data generation methodology
The large corpus was produced by the deterministic streaming generator in src/stream_10gb_batch.py. The generator combines phase-specific Hausa phrase banks with controlled values for names, places, days, times, topics, record numbers, and sample codes. It writes one JSON record at a time to gzip-compressed JSONL, so the full corpus does not need to be held in memory or stored locally at once.
Each record receives a batch-specific and phase-specific identifier. The generator records the phase, task family, domain, difficulty, input, expected output, labels, context, safety profile, split, provenance fields, and quality-check metadata. Each phase was generated until its raw UTF-8-equivalent payload reached approximately 1,000,000,000 bytes. The resulting shards were validated, uploaded, checked remotely, and then removed from local staging storage.
The initial small pilot included model-assisted expansion experiments. The 100 GB production corpus was generated using streamed_combinatorial_template_v1. This distinction matters when interpreting diversity and linguistic quality.
Curriculum phases
Record schema
Every production record follows the same broad JSON structure:
Measured metrics
The following metrics were computed from the 100 production validation manifests and a reproducible random sample of three records from each phase in Batch 010.
The validation metrics establish technical integrity, not linguistic correctness. Exact hash uniqueness should not be confused with semantic diversity: the generator embeds a unique sample code in each prompt, so near-duplicate templates can still exist. Independent Hausa-speaker review, semantic deduplication, and naturalness scoring remain necessary for a production-quality corpus.
Quality assessment and limitations
The corpus has strong technical properties: a stable schema, deterministic identifiers, reproducible batch generation, phase coverage, gzip integrity, split accounting, and machine-readable manifests. It is therefore useful for testing training infrastructure and controlled task-format experiments.
Its principal weakness is linguistic and semantic diversity. Many records are produced by recombining a finite set of templates. A changed name, location, time, topic, or record code can make a record technically different while leaving the underlying sentence pattern nearly unchanged. Some combinations may sound unnatural to a Hausa speaker, and some task labels are used as synthetic topic values. The metadata fields language_review and safety_review remain pending until qualified human reviewers assess the records.
The production release uses general Hausa Latin orthography. It should not be assumed to represent all regional varieties, dialects, registers, or Ajami writing. It also contains no claim that synthetic outputs are factual statements about the people, places, dates, or topics appearing in the records.
Recommended use in training
For responsible experimentation, use the corpus as one component of a broader mixture. Begin with a small sample and inspect language quality. Deduplicate semantically across batches, not only by exact string hash. Keep validation and test examples isolated from template families used in training. Weight the corpus alongside human-authored and community-reviewed Hausa text, and evaluate the resulting model with native-speaker judgments covering fluency, adequacy, factuality, dialect fit, harmfulness, and instruction-following.
Loading the data
The repository is available at:
<https://huggingface.co/datasets/rufatronics/hausa-10phase-synthetic-training-corpus>
The phase shards are stored beneath data/batch_001 through data/batch_010. The validation manifests are stored beneath the corresponding batch directories. Because this release is intentionally sharded and compressed, users may choose a subset of batches or phases for experiments rather than loading the full corpus at once.
License and attribution
The repository metadata declares cc-by-4.0 for the synthetic artifacts. Users should verify that license suitability for their intended deployment and should retain the dataset card and provenance information when redistributing derived artifacts.
Collection
This dataset is included in the public VelkroLM Hausa collection:
<https://huggingface.co/collections/VelkroLM/hausa>
Citation
@dataset{hausa_10phase_synthetic_training_corpus,
title = {Hausa 10-Phase Synthetic Training Corpus},
author = {Manus AI},
year = {2026},
publisher = {Hugging Face},
repository = {rufatronics/hausa-10phase-synthetic-training-corpus},
language = {Hausa},
note = {Synthetic streamed combinatorial corpus with ten curriculum phases}
}