common-pile/cccc_filtered

Creative Commons Common Crawl Description This dataset contains text from 52 Common Crawl snapshots, covering about half of Common Crawl snapshots available to date and covering all years of operations of Common Crawl up to 2024. We found a higher level of duplication across this collection, suggesting that including more snapshots would lead to a modest increase in total token yield. From these snapshots, we extract HTML content using FastWarc. Then, using a… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/cccc_filtered.