Datasetpublic

bertin-project/mc4-sampling

A sampling-enabled version of mC4, the colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org". This is a version of the processed version of Google's mC4 dataset by AllenAI, in which sampling methods are implemented to perform on the fly.

sourceHugging Faceodc-byupdated 2y agoView on Hugging Face

13likes190downloads

root

dirdummy/

file.gitattributes690 Bdownload

filemc4-sampling.py15 KBdownload

fileREADME.md11 KBdownload

bertin-project/mc4-sampling · main · files are served by the source, never re-hosted here