openeurollm/open-perfectblend-decontaminated

Decontamination

This dataset is a decontaminated version of mlabonne/open-perfectblend.

Benchmarks used

• —MATH500: HuggingFaceH4/MATH-500 (subset=default, split=test)

• —AIME24: HuggingFaceH4/aime_2024 (subset=default, split=train)

• —AIME25: math-ai/aime25 (subset=default, split=test)

• —AMC23: math-ai/amc23 (subset=default, split=test)

• —JEEBench: daman1209arora/jeebench (subset=default, split=test)

• —GPQADiamond: Idavidrein/gpqa (subset=gpqa_diamond, split=train)

• —LiveCodeBench: ali-elganzory/livecodebench-code_generation_lite (subset=release_v6, split=test)

• —HumanEval: openai/openai_humaneval (subset=openai_humaneval, split=test)

• —MBPP: google-research-datasets/mbpp (subset=full, split=train+test+validation+prompt)

• —IFEval: google/IFEval (subset=default, split=train)

• —AlpacaEval: tatsu-lab/alpaca_eval (subset=alpaca_eval, split=eval)

• —Arena-Hard-v2.0: lmarena-ai/arena-hard-auto (subset=default, split=train) (data_files=['data/arena-hard-v2.0/question.jsonl'])

Decontamination settings

<table> <thead> <tr><th>Parameter</th><th>Value</th></tr> </thead> <tbody>

<tr><td>N-gram size</td><td>8</td></tr>

<tr><td>Match threshold</td><td>0.5</td></tr>

</tbody> </table>

Split and benchmark details

<table> <thead> <tr> <th>Subset</th> <th>Split</th> <th>Docs in split (dataset)</th> <th>Benchmark</th> <th>Contaminated (dataset)</th> <th>Contamination rate (dataset)</th> <th>Docs (benchmark)</th> <th>Contaminated (benchmark)</th> <th>Contamination rate (benchmark)</th> </tr> </thead> <tbody>

<tr> <td rowspan="12">default</td> <td rowspan="12">train</td> <td rowspan="12">1,884,616</td> <td>MATH500</td> <td>803</td> <td>0.0426%</td> <td>500</td> <td>91</td> <td>18.20%</td> </tr>

<tr>

<td>AIME24</td> <td>0</td> <td>0.0000%</td> <td>30</td> <td>0</td> <td>0.0000%</td> </tr>

<tr>

<td>AIME25</td> <td>2</td> <td>0.0001%</td> <td>30</td> <td>1</td> <td>3.33%</td> </tr>

<tr>

<td>AMC23</td> <td>3</td> <td>0.0002%</td> <td>40</td> <td>2</td> <td>5.00%</td> </tr>

<tr>

<td>JEEBench</td> <td>0</td> <td>0.0000%</td> <td>515</td> <td>0</td> <td>0.0000%</td> </tr>

<tr>

<td>GPQADiamond</td> <td>8</td> <td>0.0004%</td> <td>198</td> <td>6</td> <td>3.03%</td> </tr>

<tr>

<td>LiveCodeBench</td> <td>2</td> <td>0.0001%</td> <td>1055</td> <td>2</td> <td>0.1896%</td> </tr>

<tr>

<td>HumanEval</td> <td>374</td> <td>0.0198%</td> <td>164</td> <td>117</td> <td>71.34%</td> </tr>

<tr>

<td>MBPP</td> <td>468</td> <td>0.0248%</td> <td>974</td> <td>227</td> <td>23.31%</td> </tr>

<tr>

<td>IFEval</td> <td>10</td> <td>0.0005%</td> <td>541</td> <td>5</td> <td>0.9242%</td> </tr>

<tr>

<td>AlpacaEval</td> <td>309</td> <td>0.0164%</td> <td>805</td> <td>85</td> <td>10.56%</td> </tr>

<tr>

<td>Arena-Hard-v2.0</td> <td>10</td> <td>0.0005%</td> <td>750</td> <td>6</td> <td>0.8000%</td> </tr>

</tbody> </table>

Dataset summary

<table> <thead> <tr><th>Metric</th><th>Value</th></tr> </thead> <tbody> <tr><td>Total documents in dataset</td><td>1,884,616</td></tr> <tr><td>Contaminated documents (removed)</td><td>1,984</td></tr> <tr><td>Documents after decontamination</td><td>1,882,632</td></tr> <tr><td>Contamination rate (dataset)</td><td>0.1053%</td></tr> </tbody> </table>

🎨 Open-PerfectBlend

Open-PerfectBlend is an open-source reproduction of the instruction dataset introduced in the paper "The Perfect Blend: Redefining RLHF with Mixture of Judges". It's a solid general-purpose instruction dataset with chat, math, code, and instruction-following data.

Data source

Here is the list of the datasets used in this mix:

The deduplication process removed 88.1k samples across all datasets. All of these datasets use either an Apache 2.0 or MIT license.

Thanks to OpenBMB, MetaMath, Hugging Face, Microsoft, theblackcat102, Post-training-Data-Flywheel, and LMSYS for the data!

Comparison

Here is the extract from the paper with the dataset mixture:

There are two main differences with the dataset described in the paper:

• —Instruction-following data comes from another source because Meta didn't release their dataset.
• —The harmful intent hasn't been released either, so I didn't add any data in this category.