Apppublic

openbmb/Ultra-FineWeb-L2-Selector

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face

3likes

App README

⚡ Ultra-FineWeb Classifier

A lightweight fastText-based classifier for filtering high-quality web data, supporting both English and Chinese.

🌟 Features

—Fast Inference: Based on fastText for efficient classification
—Bilingual Support: Works with both English (en) and Chinese (zh) content
—Quality Scoring: Returns a quality score from 0 to 1
—Easy to Use: Simple web interface powered by Gradio

📊 Quality Score Interpretation

Score Range	Quality Level	Recommendation
≥ 0.7	🌟 High Quality	Suitable for LLM training
0.4 - 0.7	📊 Medium Quality	May need review
< 0.4	⚠️ Low Quality	Likely not suitable

🔗 Links

📝 Citation

bibtex

@misc{wang2025ultrafineweb,
  title={{Ultra-FineWeb}: Efficient Data Filtering and Verification for High-Quality LLM Training Data},
  author={Yudong Wang and Zixuan Fu and Jie Cai and Peijun Tang and Hongya Lyu and Yewei Fang and Zhi Zheng and Jie Zhou and Guoyang Zeng and Chaojun Xiao and Xu Han and Zhiyuan Liu},
  year={2025},
  eprint={2505.05427},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
}

📄 License

Apache 2.0