ghanaopenai/ghanaian-corpus-generate

Ghanaian Corpus — Generated Content-Writer Prompts

Description

Synthetic content-writer prompts generated from 534,294 Ghanaian corpus chunks using Qwen2.5-0.5B-Instruct via vLLM.

Each row contains:

• —source_type: Category of the source document (e.g., parliament, news, academic)
• —source: Name or identifier of the source document
• —page_range: Page range within the source
• —text: The original corpus chunk text (minimum 4 sentences, each with 6+ unique alphabetic words)
• —generated_question: A synthetic prompt representing what a content writer would type into an AI assistant to produce text similar to the given chunk

How it was built

1. 1.Corpus collection: ~1.16M text chunks from Ghanaian parliament hansards, news articles, academic papers, and books
2. 2.Quality filtering: Chunks with fewer than 4 sentences or sentences with fewer than 6 unique alphabetic words (excluding numbers) were dropped — retaining 534,294 chunks
3. 3.Prompt generation: Qwen2.5-0.5B-Instruct served via vLLM (128 concurrent workers) generated one content-writer prompt per chunk
4. 4.Cleaning: Multi-line boilerplate prefixes removed, surrounding quotation marks stripped

Intended Use

Fine-tuning LLMs to generate text grounded in Ghanaian context. The text column provides the target content while generated_question provides the instruction-style prompt, making this suitable for instruction-tuning or supervised fine-tuning.

License

Same as the original Ghanaian Research Corpus.