Athekunal/english-hindi-reasoning-dataset

english-hindi-reasoning-dataset

English→Hindi translation dataset for chain-of-thought reasoning, segmented into PRM-style reasoning steps. Built for training a masked diffusion language model (MDLM) to translate reasoning traces step-by-step.

Sources

Reasoning traces were sampled from three upstream datasets and translated English→Hindi at the reasoning-step level (each step: one coherent chunk of a chain-of-thought, bounded by heading/discourse-marker/semantic-break detection). The source field on every document records exactly which upstream dataset it came from.

• —`open-thoughts/OpenThoughts3-1.2M` — diverse-domain CoT
• —`facebook/natural_reasoning` — general reasoning
• —`nvidia/OpenCodeReasoning` — code reasoning

Splits

Train

Validation

Split is deterministic per-document (md5 hash of doc_id), so a document's steps never cross the train/val boundary.

Schema

Each row is one document:

{
  "doc_id": "md5 hash of the original English question, stable across reruns",
  "source": "full HF dataset id the document was sampled from",
  "question": "original English question/prompt",
  "num_steps": "int, number of reasoning steps",
  "steps": [
    {
      "step_index": "int, 0-based order within the document",
      "boundary_reason": "why this step starts here (heading / discourse marker / semantic break / token budget)",
      "token_count": "int, English token count for this step",
      "en": "English step text",
      "hi": "Hindi translation of this step",
      "has_missing_translation": "bool, true if translation was unavailable for this step"
    }
  ]
}

Math, code, numbers, and links inside each step are protected during translation and reinserted as <placeholder-N> tags in both en and hi — a translation model is expected to reproduce the tag verbatim in its Hindi output, and the original content is spliced back in afterward.

sft config (sft_split split)

100,000 chunked English→Hindi reasoning steps (12,596 documents) used for SFT of the translation models, with the "Wait" mistranslation post-fix applied. It is a separate config because its schema is smaller than the default config: id (string) and steps, a list of {en, hi} pairs.

from datasets import load_dataset
ds = load_dataset("Athekunal/english-hindi-reasoning-dataset", "sft", split="sft_split")