mkd-minju/keural-v2-self-verification

Self-Verification Dataset

Status: generation in progress. 4,477 / 50,000 target rows (~9%), growing. Being generated in parallel across multiple environments/models — see Generation models mix below. This card describes the file as of this snapshot; row count and model mix will change on re-upload.

File: 05_self_verification_generated.jsonl (one JSON object per line).

What this is

Synthetic "wrong draft → self-critique → corrected solution" traces, for training a model to catch and fix its own reasoning mistakes. Each row shows a model:

1. 1.Solving a problem with a specific, plausible, intentional error that leads to a wrong final answer
2. 2.Critiquing its own draft, identifying exactly where and why it went wrong
3. 3.Producing a corrected solution that arrives at the actual correct answer

Full generation spec (self-contained, written so a fresh environment/model can reproduce this independently): SELF_VERIFICATION_HANDOFF.md in the parent project root.

Why this exists

No dataset — real or synthetic — was found with this exact "draft → catch your own error → correct" structure natively. The closest prior art only covers part of it: openai/prm800k (step-level correctness labels, would need restructuring into draft/correction pairs), nvidia/OpenMathReasoning's GenSelect subset (picks the best of several candidates — a different pattern from revising one draft), and prometheus-eval/Feedback-Collection / HAERAE-HUB/K2-Feedback (critique+score structure, no correction field). See DATASET_PLAN.ko.md §1.5 for the full survey.

Conversation structure

{
  "source": "self-generated",
  "license": "cc-by-4.0",
  "share_alike": false,
  "lang": "en",
  "category": "self_verification",
  "conversations": [
    {
      "role": "user",
      "content": "<problem text>",
      "reasoning_content": null,
      "tool_calls": []
    },
    {
      "role": "assistant",
      "content": "<corrected final solution>",
      "reasoning_content": "<wrong draft>\n\nWait, let me double check this.\n\n<self-critique>",
      "tool_calls": []
    }
  ],
  "meta": {
    "draft": "<wrong draft, preserved standalone for audit>",
    "critique": "<self-critique, preserved standalone>",
    "correction": "<corrected solution, preserved standalone>",
    "verified_wrong": true,
    "generation_model": "string",
    "prompt_template": "self_verification_draft_critique_correction_v1",
    "seed_source": { "source_dataset": "04_cot_sampled_50000", "source_index": 0 }
  }
}

Only 2 conversation turns per row (user = problem, assistant = final answer) — the draft-critique-correction structure lives inside the single assistant turn: reasoning_content holds the wrong draft + critique (destined for <think>...</think> at final encoding time), content holds only the clean corrected answer. meta.draft/meta.critique/meta.correction additionally preserve all three stages as separate strings, for auditing independent of how they're concatenated into reasoning_content.

Every field is uniform across all 4,477 rows except meta.generation_model: source="self-generated", license="cc-by-4.0", share_alike=false, lang="en", category="self_verification", meta.verified_wrong=true (rows that failed verification are never written — see Verification), meta.prompt_template="self_verification_draft_critique_correction_v1", meta.seed_source.source_dataset="04_cot_sampled_50000".

Turn-level content length in this snapshot: draft averages 1,739 chars (median 1,612, max 8,126), critique 1,518 chars (median 1,390, max 6,402), correction 2,154 chars (median 1,948, max 14,843) — corrections run somewhat longer than drafts, consistent with a corrected solution re-deriving the answer rather than just patching one step. Problem text (user turn) averages 186 chars (median 161, max 853).

Seeding

Seeds are problem + known-correct-answer pairs, not open-ended prompts — this is a hard requirement of the category, not a design preference: verifying "is this draft actually wrong" and "is this correction actually right" needs a ground-truth answer to check against, which most conversational data (e.g. §1.1 fluency) doesn't have.

• —Source: mkd-minju/keural-v2-cot-reasoning, file 04_cot_sampled_50000.json (underlying data: nvidia/OpenMathReasoning, CC-BY-4.0, olympiad-level math problems with solutions from DeepSeek-R1/QwQ-32B)
• —Only rows with both a non-empty problem and expected_answer are used as seeds (~49,984 of the 50,000)
• —100% of seeds so far are from `04_cot_sampled_50000` — all math. This category is defined generically ("self-verification") but in practice, math is the only seed source currently plugged in, because it's the only upstream category with checkable ground-truth answers. If broader (non-math) self-verification is wanted later, it needs a different seed source with the same problem+answer structure.
• —One seed is used at most once (no retries with the same seed after a rejection or failure) — the seed pool (~49,984) comfortably covers the 50,000 target with margin for the rejection/failure rate below.

Generation method

Each seed goes through, in order:

1. 1.Wrong draft: model is explicitly instructed to introduce "a specific, plausible reasoning mistake" leading to a final answer different from the known correct one, without revealing the mistake
2. 2.Verify actually wrong: the draft's stated answer (extracted only from an explicit "answer:" / "final answer:" phrase — never guessed from trailing text, to avoid false-positive matches on short numeric answers) is compared against the known answer; if ambiguous, an LLM judge call decides MATCH/MISMATCH. If the draft accidentally landed on the correct answer anyway, the seed is discarded (not retried) — an accidentally-correct "wrong" draft would teach the opposite of the intended lesson
3. 3.Self-critique: model reviews its own draft and must identify the specific error
4. 4.Correction: model writes a corrected solution using the critique
5. 5.Verify actually correct: same check as step 2, against the correction — if the correction still doesn't match the known answer, the seed is discarded

Only seeds passing both checks are written. In the local generation run (2,768 rows since resuming from a 1,710-row checkpoint), the resolved-attempt breakdown was roughly 61% written / 18% rejected (wrong-not-actually-wrong or correction-not-actually-correct) / 21% failed (API/generation errors) — within the 60–90% success-rate range anticipated in the handoff spec, at the lower end.

Generation models mix (important)

Unlike the other synthetic categories in this project (§1.3 used one model throughout), this dataset is being assembled from multiple environments/models in parallel — the handoff doc was written specifically to let independent environments contribute compatible rows:

All open-weight, satisfying the project's no-closed-API policy (avoids competing-model ToS restrictions on using outputs to train other models). Expect stylistic/difficulty variance across rows as a direct consequence of the mixed models — filter or stratify by meta.generation_model if that variance matters for your use case. Row-level model attribution means you can always recover which rows came from which source.

Verification, not just generation

This is the one category in the project with a built-in adversarial check baked into the generation loop itself rather than deferred to a later manual review pass: both the "is the draft actually wrong" and "is the correction actually right" checks run automatically during generation, and non-passing attempts are discarded before ever being written. meta.verified_wrong: true is a hard filter — it will be true on literally every row in this file because rows that failed the check are never written in the first place (this is enforced in code, not just documented). A separate, still-pending check (per DATASET_PLAN.ko.md §4) is a native-speaker/second-party adversarial review to confirm the regex+LLM-judge automated verification wasn't itself fooled (e.g. by a judge that itself mis-grades) — that's a quality gate on top of this one, not yet run.

Known failure mode this generator specifically guards against

Olympiad-level problems induce long model reasoning; with an insufficient token budget, generation gets cut off mid-answer rather than completing. If that truncated, incomplete text were accepted as a valid draft/critique/correction, it would silently corrupt the dataset (and can also fool a judge model into a false MATCH/MISMATCH verdict on an unfinished answer). This generator requires max_tokens >= 6000 and timeout >= 500s on the three generation calls, and explicitly checks the API's finish_reason — a response with finish_reason == "length" is treated as an error and discarded, regardless of whether content looks superficially complete. See SELF_VERIFICATION_HANDOFF.md §5.1 for the full incident writeup this defense came from.

Pipeline stage (important)

Like the other synthetic categories in this project, this has only completed generation — none of DATASET_PLAN.ko.md §3's processing has been run yet:

• —❌ No deduplication
• —❌ No length/tokenizer-distribution check against the real target tokenizer
• —❌ No train/val/test split
• —❌ No final chat-template encoding — role/content kept raw on purpose (the target model's <think>...</think> encoding is applied project-wide at the end of §3, not per-category)
• —❌ No contamination cross-check against eval benchmarks yet
• —⏳ §4's adversarial native-speaker review of the auto-verification (see above) not yet run
• —✅ Per-row provenance (meta.generation_model, meta.prompt_template, meta.seed_source) complete

Known limitations

• —100% math seeds (see Seeding) — despite the category name, this is not currently a general logical-reasoning self-correction dataset, it's specifically math self-correction
• —Mixed generation models (4 distinct so far) — see Generation models mix
• —Partial/in-progress — everything above (counts, model-mix ratios) will shift as generation continues
• —Private repo — internal use only for now

License

Fully self-generated content (only the seed problem/expected_answer pairs come from nvidia/OpenMathReasoning, CC-BY-4.0) — tagged cc-by-4.0 to match the seed license and the other self-generated categories in the parent collection.

Provenance

Part of the Korean SFT dataset collection project (mkd-minju), §1.5 of DATASET_PLAN.ko.md. Reproduction spec: SELF_VERIFICATION_HANDOFF.md. Generation script: data-repo/processed/05_self_verification/generate_05_self_verification.py. Formal schema: data-repo/schema/korean_sft_schema.md §3.5.