ghanaopenai/twi-english-reasoning-translation

Twi-English Reasoning Translation Dataset

Chain-of-thought English<->Twi (Akan) translation data, generated for fine-tuning MiniCPM5-1B. Each row is a two-turn chat conversation: the user asks for a translation, the assistant answers with its reasoning inside <think>...</think> (matching MiniCPM5-1B's chat template, which splits assistant content on those tags) followed by the translation itself.

Sources

For the three "gold reference" sources, Gemini is given the source sentence AND its existing correct translation, and asked only to explain the grammatical/idiomatic reasoning connecting them -- it never generates or alters the translation itself, so those outputs can't drift from the vetted parallel text. Only the asr source has Gemini generate a real translation, since no reference exists for those transcripts.

Markdown augmentation

Roughly 70% of rows have 2-3 stacked Markdown techniques applied to the SAME sentence(s) on both the source and translated text, so the model learns to carry Markdown formatting through translation instead of stripping it:

• —a heading on its own line (## sentence) on almost every augmented row -- about 1/3 of those are rendered in ALL CAPS, the rest normal case
• —one body-wide structure on top: numbered list, bullet list, table, or bold paragraph
• —0-2 inline spot styles (bold or italic) on individual sentences, which can nest inside an already-listified body

Format

{
  "id": "...",
  "source": "pristine | ghana_chat_en2tw | ghana_chat_tw2en | asr",
  "markdown": "heading,numbered_list,bold_sentence  (or null if unaugmented)",
  "messages": [
    {"role": "user", "content": "Translate the following English text into Twi (Akan). Return only the translation with no additional text.\n\n<text>"},
    {"role": "assistant", "content": "<think>\n<reasoning>\n</think>\n\n<translation>"}
  ]
}

Generation scripts

The exact scripts used to build this dataset are included under scripts/:

• —sources.py -- samples/extracts source text from the three underlying HF datasets (column-projected reads, so audio/unused columns are never downloaded)
• —pipeline.py -- async Gemini-based translation + reasoning generation engine, rate-limited and resumable
• —markdown_augment.py -- the Markdown-mirroring augmentation described above
• —push_conversations.py -- converts generated rows into the chat-format conversations above and pushes them to the Hub
• —run_full.py -- orchestrates the full run across all four sources concurrently, pushing an updated snapshot to this repo every 5,000 new rows

Reasoning/translation generation model: Gemini (gemini-3.6-flash), via the google-genai SDK.