agentlans/epfml-FineWeb2-HQ-sample
epfml/FineWeb2-HQ A curated subset of the epfml/FineWeb2-HQ dataset featuring high-quality multilingual text. Details First 25 000 rows per config (language and script pair) Duplicates removed Texts truncated to 512 LLaMA 3.1 tokens Scores transformed with log10 Rows shuffled and 20% of the rows split into the test set (stratified by config) Example { "text": "爵士大师Tim Garland 深圳专场 - [jazz]\nTim Garland Lighthouse… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/epfml-FineWeb2-HQ-sample.
051
1---2task_categories:3- text-generation4language:5- ru6- zh7- de8- ja9- es10- fr11- it12- pt13- pl14- nl15- id16- tr17- cs18- vi19- sv20- fa21- ar22- el23- da24- hu25license: odc-by26tags:27- multilingual28- quality29---30# epfml/FineWeb2-HQ31 32A curated subset of the [epfml/FineWeb2-HQ](https://huggingface.co/datasets/epfml/FineWeb2-HQ) dataset featuring high-quality multilingual text.33 34## Details35 36- First 25 000 rows per config (language and script pair)37- Duplicates removed38- Texts truncated to 512 LLaMA 3.1 tokens39- Scores transformed with log1040- Rows shuffled and 20% of the rows split into the test set (stratified by config)41 42## Example43 44```json45{46 "text": "爵士大师Tim Garland 深圳专场 - [jazz]\nTim Garland Lighthouse Trio\n来自英国的著名爵士萨克斯风演奏家，作曲家，乐团领队。同时他的低音黑管演奏和大量的作曲作品在世界范围内享有盛名。他是一个全面而勇于创新的音乐家，他模糊和爵士乐和古典音乐的边界，使得音乐有了无数的可能性。\n作为乐手，他长期与爵士音乐大师Chick Corea合作并获得过格莱美奖。是当今世界上最炙手可热的爵士音乐家之一。\nhttp://www.douban.com/event/11121480/",47 "config": "cmn_Hani",48 "log10_score": -0.870128716695649749}50```51 52## License53 54Open Data Commons License Attribution