liodon-ai/oasst1-decontaminated

Decontaminated — OpenAssistant/oasst1

What this is

A filtered version of OpenAssistant/oasst1 (revision fdf72ae0827c1cda404aff25b6603abec9e3399b) with exact-duplicate rows and rows overlapping standard benchmark test sets removed. This is a different artifact from the companion contamination report — that one is an audit of what's wrong; this one is the corpus with those rows actually taken out, ready to train on.

Processing

1. 1.Deduplicated exact-match rows (normalized-text hash).
2. 2.Removed rows with any 13-gram overlap against MMLU/GSM8K/HellaSwag/HumanEval test sets (same methodology as the companion contamination report — see that dataset card for details).
3. 3.Split into train/validation via a deterministic content-hash bucket (98/2), so no row can appear in both splits.

Statistics

• —Source rows: 84,437
• —Removed as exact duplicates: 1,030
• —Removed as benchmark-contaminated: 2
• —Remaining rows: 83,405
• —Training split: 81,795
• —Validation split: 1,610
• —Benchmarks checked: gsm8k, hellaswag, humaneval, mmlu

Limitations

• —Only exact-duplicate and exact-n-gram-overlap removal — near-duplicates (reworded/recompressed copies) and paraphrased benchmark leakage are not caught. See the companion contamination report for the full per-row audit this filtering was based on.

License

Inherits the source dataset's license (apache-2.0).

Citation

@misc{liodonai_oasst1_decontaminated,
  title        = {Decontaminated — oasst1},
  author       = {{Liodon AI}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/liodon-ai/oasst1-decontaminated}},
  note         = {Deduplicated and decontaminated derivative of OpenAssistant/oasst1}
}

Generated by Liodon AI's automated dataset-factory pipeline