hkust-nlp/PreSelect-100B
📑 Paper | 🔨 fastText Classifier | 🤗 Released Dataset | 📦 Repo PreSelect-100B is a curated ~100B token pretraining dataset that achieves great performance on various benchmarks. It is filtered by PreSelect-Classifier at 10% threshold, where the pool is a randomly sampled subset of DCLM-refinedweb, which is a cleaned version of Common Crawl raw data but without any model-based filtering.… See the full description on the dataset page: https://huggingface.co/datasets/hkust-nlp/PreSelect-100B.
<p align="center"> 📑 <a href="https://arxiv.org/abs/2503.00808" target="blank">Paper</a>    |    🔨 <a href="https://huggingface.co/hkust-nlp/preselect-fasttext-classifier" target="blank">fastText Classifier</a>    |    🤗 <a href="https://huggingface.co/datasets/hkust-nlp/PreSelect-100B" target="blank">Released Dataset</a>    |    📦 <a href="https://github.com/hkust-nlp/PreSelect" target="blank">Repo</a> <br> </p>
PreSelect-100B is a curated ~100B token pretraining dataset that achieves great performance on various benchmarks. It is filtered by PreSelect-Classifier at 10% threshold, where the pool is a randomly sampled subset of DCLM-refinedweb, which is a cleaned version of Common Crawl raw data but without any model-based filtering.
Benchmark results
Trianing using PreSelect curated dataset achieve superior results than other dataset selection methods on various downstream tasks and below are comparisons.
Citation
If you find this work helpful, please kindly cite as:
@article{shum2025predictivedataselectiondata,
title={Predictive Data Selection: The Data That Predicts Is the Data That Teaches},
author={Kashun Shum and Yuzhen Huang and Hongjian Zou and Ding Qi and Yixuan Liao and Xiaoxin Chen and Qian Liu and Junxian He},
journal={arXiv preprint arXiv:2503.00808},
year={2025},
eprint={2503.00808},
}