Datasetpublic

kilicai/turkish-sft-clean-v5-strict-pruned

Clean Turkish SFT v5 Strict Pruned This is the strictest cleaned version. It prioritizes correctness over row count: content errors, mislabeled rows, unsafe/refusal-contaminated examples, exact duplicates, and near-duplicates in template-heavy categories were deleted rather than padded. What was fixed Removed category-contaminated rows with task-specific validators. Removed exact duplicates by normalized SHA-256 fingerprint. Removed near-duplicates in… See the full description on the dataset page: https://huggingface.co/datasets/kilicai/turkish-sft-clean-v5-strict-pruned.

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes23downloads

Dataset Card

Clean Turkish SFT v5 Strict Pruned

This is the strictest cleaned version. It prioritizes correctness over row count: content errors, mislabeled rows, unsafe/refusal-contaminated examples, exact duplicates, and near-duplicates in template-heavy categories were deleted rather than padded.

What was fixed

—Removed category-contaminated rows with task-specific validators.
—Removed exact duplicates by normalized SHA-256 fingerprint.
—Removed near-duplicates in formal-writing, creative-writing, and multi-turn-dialogue using normalized prompt/answer-prefix fingerprints.
—Removed explicit unsafe/toxic/refusal-contaminated content.
—Verified all remaining CoT rows have #### answer matching metadata when present.
—Enforced real multi-turn structure for multi-turn-dialogue.

Audit

json

{
  "rows": 76276,
  "categories": {
    "cot": 14388,
    "instruction-following": 9987,
    "qa": 9082,
    "summarization": 7657,
    "code-explanation": 7425,
    "translation": 6580,
    "error-correction": 6066,
    "text-classification": 5468,
    "formal-writing": 5032,
    "creative-writing": 3811,
    "rewriting": 689,
    "multi-turn-dialogue": 91
  },
  "sources": {
    "v4": 50257,
    "generated_verified_math_v2_utf8": 14388,
    "synthetic_verified_topup_v3": 6905,
    "synthetic_qa": 4445,
    "synthetic": 248,
    "synthetic_verified_multiturn_v4": 33
  },
  "drop_stats": {
    "drop_translation_no_marker": 3394,
    "drop_error_no_marker": 3900,
    "drop_multiturn_not_multiturn": 5631,
    "drop_rewrite_no_marker": 9292,
    "drop_summ_no_prompt_marker": 2302,
    "drop_code_no_code": 2563,
    "drop_formal_no_formal_markers": 4584,
    "drop_class_no_marker": 4487,
    "drop_creative_no_marker": 6188,
    "drop_near_duplicate": 4638,
    "drop_safety_refusal": 32,
    "drop_qa_no_question": 811
  },
  "bad_format_or_category": 0,
  "exact_duplicates": 0,
  "near_duplicates_keyed": 0,
  "safety_hits": 0,
  "cot_bad_answers": 0,
  "multi_turn_valid_rows": 91
}

Usage

python

from datasets import load_dataset
ds = load_dataset("kilicai/turkish-sft-clean-v5-strict-pruned", split="train")

For maximum volume use kilicai/turkish-sft-clean-v4-final; for strictest correctness use this repo.

Generated by ML Intern

This dataset repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

—Try ML Intern: https://smolagents-ml-intern.hf.space
—Source code: https://github.com/huggingface/ml-intern