stindardlogic/chain-of-thought-dpo-2k

Chain-of-Thought DPO Pairs (2.6K)

DPO preference pairs for training LLMs to reason explicitly before answering.

Dataset Description

2,600 preference pairs across 6 reasoning categories:

Chosen responses always include:

• —Explicit numbered reasoning steps
• —Intermediate calculations/derivations shown
• —Clear final answer

Rejected responses include 4 failure modes:

• —bare_answer — correct answer, no reasoning shown
• —wrong_answer — plausible but incorrect answer
• —shallow_reasoning — only first step, then jumps to conclusion
• —confident_wrong — confidently states incorrect answer

Format

{
  "prompt": "A train leaves Station A at 9:00 AM...",
  "chosen": "Let me think through this step by step.\n\n**Step 1:** Define the meeting point...\n...\n**Answer:** The trains meet at approximately 11:43 AM.",
  "rejected": "The trains meet after about 2 hours, so around 11:00 AM.",
  "metadata": {
    "category": "math_word",
    "rejection_type": "wrong_answer",
    "reasoning_steps": 7
  },
  "id": "abc123def456"
}

Use Case

• —Train models to use chain-of-thought reasoning via DPO/RLHF
• —Teach step-by-step decomposition before answering
• —Reduce confident hallucination on reasoning tasks
• —Compatible with DPO, IPO, KTO frameworks

Related Datasets

• —`stindardlogic/dpo-reasoning-70k` — Larger reasoning DPO dataset

License

Apache 2.0