Datasetpublic

mkurman/trlm-dpo-stage-3-synth

trlm-dpo-stage-3 (synth reasoning rewrite) Direct Preference Optimization (DPO) dataset pairing original DeepSeek-R1 distillation responses against synth-style reasoning rewrites produced by DeepSeek V4 Flash. Source The rejected side originates from Shekswess/trlm-dpo-stage-3-final-2. Each original assistant response followed the DeepSeek-R1-Distill style <think>...</think>\n\n<final answer> layout. For each record the <think> block was stripped of its tags to… See the full description on the dataset page: https://huggingface.co/datasets/mkurman/trlm-dpo-stage-3-synth.

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes30downloads

Dataset Card

trlm-dpo-stage-3 (synth reasoning rewrite)

Direct Preference Optimization (DPO) dataset pairing original DeepSeek-R1 distillation responses against synth-style reasoning rewrites produced by DeepSeek V4 Flash.

Source

The rejected side originates from `Shekswess/trlm-dpo-stage-3-final-2`. Each original assistant response followed the DeepSeek-R1-Distill style <think>...</think>\n\n<final answer> layout. For each record the <think> block was stripped of its tags to recover reasoning_content, and the trailing text after </think> was kept as content.

The chosen side was generated by DeepSeek V4 Flash, which rewrote both the reasoning (into a compact, structured "synth-like" format) and the final answer.

Format

Each record has three fields, each a list of chat messages:

json

{
  "prompt":   [{"role": "user", "content": "..."}],
  "chosen":   [{"role": "assistant", "reasoning_content": "<synth reasoning>", "content": "<rewritten answer>"}],
  "rejected": [{"role": "assistant", "reasoning_content": "<original reasoning>",      "content": "<original answer>"}]
}

—prompt — shared user turn (identical for chosen and rejected).
—chosen.reasoning_content / chosen.content — DeepSeek V4 Flash rewrite.
—rejected.reasoning_content — original chain-of-thought with <think> tags removed.
—rejected.content — original final answer (text after </think>).

Splits

split	examples
train	25,683
eval	1,351
total	27,034

Records are shuffled (seed 42) and split 95 / 5.

Intended use

DPO / ORPO / KTO training of reasoning models that should prefer compact, well-structured chain-of-thought over verbose unstructured reasoning.

Reasoning length: synth vs. R1

The synth rewrite is dramatically more compact than the original R1-distill reasoning. Measured on reasoning_content (tags stripped on both sides) across all 27,034 pairs:

metric	synth (chosen)	R1 (rejected)	change
mean chars	1,517	7,187	−78.9 %
median chars	1,391	3,473
mean words	232	1,239	−81.3 %
median words	214	572
p90 chars	2,322	19,014
total chars (all pairs)	41.0 million	194.3 million	−78.9 %

—Synth reasoning is shorter than its R1 counterpart in 90.6 % of records (24,488 / 27,034).
—Mean chosen / rejected char ratio = 0.48, median 0.39.
—R1 reasoning has a very heavy right tail (max 100k chars, p90 ≈ 19k); synth reasoning stays tightly bounded (max ≈ 19k, p90 ≈ 2.3k) while preserving the structured argumentation needed for the task.

In short: the synth side teaches the model to reach the same conclusions with roughly ¼ the reasoning tokens of the R1-distill baseline.

Notes

—Every original record contained exactly one <think> and one </think> tag; no malformed splits.
—reasoning_content is stored without <think> tags — add them at the tokenizer/template layer if your chat template expects them.
—No deduplication was applied on top of the source dataset.