liodon-ai/oasst1-contamination-report

Contamination Report — OpenAssistant/oasst1

What this is

A row-level audit of OpenAssistant/oasst1 (revision fdf72ae0827c1cda404aff25b6603abec9e3399b) for exact 13-gram overlap with standard benchmark test sets (gsm8k, hellaswag, humaneval, mmlu). This is not a filtered copy of the source — it's a new artifact: a list of which rows overlap which benchmark, plus summary statistics, so anyone training on the source can decide how to handle it.

Methodology

1. 1.Built a combined n-gram index (13-word, lowercased, whitespace-normalized) from the test/ validation splits of: MMLU (cais/mmlu), GSM8K (openai/gsm8k), HellaSwag (Rowan/hellaswag), HumanEval (openai/openai_humaneval).
2. 2.Scanned every row of the source dataset's text field for any 13-gram appearing in that index.
3. 3.A row is flagged "contaminated" if at least one 13-gram matches — this is deliberately conservative (exact match only, not fuzzy/semantic similarity), so it has effectively zero false positives but will miss paraphrased contamination.

This is the same n-gram-overlap convention used by GPT-3 (Brown et al. 2020) and RedPajama's own decontamination reports, so these numbers are comparable to other published contamination audits.

Results

• —Source rows scanned: 84,437
• —Rows flagged as contaminated: 2 (0.002%)
• —Benchmarks checked: gsm8k, hellaswag, humaneval, mmlu

Schema

Limitations

• —Exact n-gram matching only — paraphrased or reworded benchmark leakage is not detected.
• —Only checks the four benchmarks listed above, not the full range of benchmarks a given model might be evaluated on.
• —A flagged row does not necessarily invalidate all of a source dataset — it identifies exactly which rows to review or exclude.

License

Inherits the source dataset's license (apache-2.0) — this report contains references to and excerpts from the source, not a redistribution of the full source data.

Citation

@misc{liodonai_oasst1_contamination_report,
  title        = {Contamination Report — oasst1},
  author       = {{Liodon AI}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/liodon-ai/oasst1-contamination-report}},
  note         = {Benchmark-contamination audit of OpenAssistant/oasst1}
}

Generated by Liodon AI's automated dataset-factory pipeline