alkaid0618/ChineseWebText

1# ChineseWebText: Large-Scale High-quality Chinese Web Text Extracted with Effective Evaluation Model2 3This directory contains the ChineseWebText dataset, and the EvalWeb tool-chain to process CommonCrawl Data. Our EvalWeb tool is publicly available on  github https://github.com/CASIA-LM/ChineseWebText.4 5# ChineseWebText 6 7- ### Dataset Overview8 9We release the latest and largest Chinese dataset **ChineseWebText**, which consists of **1.42 TB**  data and each text is assigned a quality score, facilitating LLM researchers to select data according to a new quality threshold. We also release a much **Cleaner subset** of **600 GB** Chinese texts with quality exceeding **90%** .10 11<div align=center><img src="./pictures/Overview_of_output_datasets.png" style="zoom:67%;" /></div>12 13- ### Data Example14 15  ```json16  {17  	"title": "潍坊银行2021年上半年净利润同比增长29.57% 不良率降至1.10%_财经_中国网",18      "score": 0.95,19      "text": "潍坊银行2021年上半年净利润同比增长29.57% 不良率降至1.10%\n中国网财经8月24日讯 潍坊银行昨日披露2021年二季度信息报告显示，截至2021			年6月末，潍坊银行资产总额1920.44亿元，较上年末增长9.34%；负债总额1789.16亿元，较上年末增长10.54%。2021年上半年，潍坊银行实现净利润			6.09亿元，同比增长29.57%。\n资产质量方面，截至2021年6月末，潍坊银行不良贷款率1.10%，较上年末下降0.13个百分点。\n资本金方面，截至				2021年6月末，潍坊银行资本充足率、核心一级资本充足率、一级资本充足率分别为11.66%、7.89%、10.13%，分别较上年末下降1.89、0.89、1.15			个百分点。",20      "url": "http://finance.china.com.cn/news/special/2021bnb/20210824/5638343.shtml",21      "source\_domain": "finance.china.com.cn"22  }23  ```24 25  - "title":  【string】The title of the data text.26  - "score":  【float】Quality score generated by the quality evaluation model.27  - "text":  【string】Text content of data sample.28  - "url":  【string】External URL, points to the original web address of the text.29  - "source_domain": 【string】The domain name of the source website.30 31# Citation32Please cite the paper if you use the data in this repo.33```shell34@misc{chen2023chinesewebtext,35      title={ChineseWebText: Large-scale High-quality Chinese Web Text Extracted with Effective Evaluation Model}, 36      author={Jianghao Chen and Pu Jian and Tengxiao Xi and Dongyi Yi and Qianlong Du and Chenglin Ding and Guibo Zhu and Chengqing Zong and Jinqiao Wang and Jiajun Zhang},37      year={2023},38      eprint={2311.01149},39      archivePrefix={arXiv},40      primaryClass={cs.CL}41}42```