ghanaopenai/twi-english-reasoning-sft-mix

Twi-English Reasoning SFT Mix

A ready-to-train supervised fine-tuning dataset for English<->Twi (Akan) translation, combining a chain-of-thought reasoning pool with a much larger direct-translation pool into ONE shuffled file, so a standard trainer/ dataloader sees the right mix without any extra sampling code. The messages format with <think>...</think> in the assistant turn works with any model whose chat template splits on those tags (or that's simply trained to treat them as plain text) -- it isn't tied to a specific model.

Why mixed like this, not trained sequentially

Training on the large direct-translation set and then fine-tuning on the reasoning set (or vice versa) causes catastrophic forgetting: the much larger direct-pair set heavily penalizes the model for emitting long <think> tokens when a short direct translation would do, so a sequential curriculum makes the model unlearn the reasoning habit almost immediately.

Instead, both pools are combined into a single training pool, with the reasoning pool given enough real representation to stay consistently present despite being much smaller than the direct pool -- see "Composition" below for how that representation was built without resorting to duplicating rows.

Composition

• —Direct pool (400,000 rows): the unused remainder of ghananlpcommunity/pristine-twi-english (273,700 rows) and michsethowusu/ghana-chat-corpus-ak (126,300 rows, i.e. every row of that corpus not already used for the reasoning pool below). Each row's translation direction (English->Twi or Twi->English) is assigned at random, since the source is gold English<->Twi parallel text either way.

The ghana-chat-corpus-ak remainder is small enough to use in full, so it needed no special sampling. The pristine-twi-english remainder (929,497 rows) is far larger than the 273,700-row budget, and a plain random fraction of a corpus that size tends to silently drop rare vocabulary entirely (a word appearing in only 1 row has roughly a 70% chance of being excluded from a ~30% random sample). To avoid that, every word appearing in 2 or fewer rows across the whole 929,497-row remainder is guaranteed a covering row via greedy set-cover before the rest of the budget is filled with plain random sampling (which already covers common words with high probability on its own).

• —Reasoning pool (113,802 rows, no duplication): built from the 38,130 real rows in ghananlpcommunity/twi-english-reasoning-translation (pristine, ghanachaten2tw, ghanachattw2en gold-reference reasoning, plus fully-generated ASR transcription reasoning, including its ~70% Markdown-mirroring augmentation). Rather than inflating this pool by duplicating identical rows, each row's reasoning was generated one item per sentence, so it can be split cleanly at sentence boundaries: every row with 2+ sentences and a clean reasoning/sentence-count match contributes its full form and two half-length variants (shorter input, shorter matching reasoning, shorter output) -- 37,836 of 38,130 rows split this way, turning them into 113,508 distinct samples; the remaining 294 rows (single-sentence or a count mismatch) stay as one whole sample. Every variant is a genuinely different token sequence, not a copy, and the mix of full-length and half-length reasoning examples also teaches the model that reasoning length should scale with input length, rather than always producing one fixed-size <think> block regardless of how short the input is.

This reasoning-translation source dataset is itself a partial run (38,130 of a planned 229,291 rows) paused on a Gemini spend cap; this mix can be rebuilt with more real reasoning rows once that run resumes.

Format

{
  "id": "...",
  "source": "pristine | ghana_chat_en2tw | ghana_chat_tw2en | asr | pristine_direct | ghana_chat_direct",
  "type": "reasoning | direct",
  "markdown": "heading,numbered_list,bold_sentence  (or null)",
  "messages": [
    {"role": "user", "content": "Translate the following English text into Twi (Akan). Return only the translation with no additional text.\n\n<text>"},
    {"role": "assistant", "content": "<think>\n<reasoning>\n</think>\n\n<translation>"}
  ]
}

For type: "direct" rows, messages[1].content is just the translation -- no <think> block at all, since that pool is meant to teach fast direct translation.

Generation scripts

The exact scripts used to build this dataset are included under scripts/:

• —sources.py -- samples the reasoning-pool sources and the unused-remainder direct-pool sources from the three underlying HF datasets (column-projected reads, audio/unused columns never downloaded; vocabulary-coverage sampling for the large pristine remainder)
• —pipeline.py -- async Gemini-based translation + reasoning generation engine used to build the reasoning pool, rate-limited and resumable
• —markdown_augment.py -- the Markdown-mirroring augmentation applied to the reasoning pool
• —push_conversations.py -- converts generated reasoning rows into chat conversations
• —run_full.py -- orchestrates the reasoning-pool generation run
• —build_final_training_set.py -- builds THIS dataset: splits the reasoning pool at sentence boundaries, combines with the direct pool, shuffles, and pushes as sharded Parquet

Reasoning/translation generation model: Gemini (gemini-3.6-flash), via the google-genai SDK.