Datasetpublic

bertin-project/mc4-sampling

A sampling-enabled version of mC4, the colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org". This is a version of the processed version of Google's mC4 dataset by AllenAI, in which sampling methods are implemented to perform on the fly.

sourceHugging Faceodc-byupdated 2y agoView on Hugging Face

13likes195downloads

root.gitattributes

.gitattributes17 linesDownload Raw Back to root

1*.bin.* filter=lfs diff=lfs merge=lfs -text2*.lfs.* filter=lfs diff=lfs merge=lfs -text3*.bin filter=lfs diff=lfs merge=lfs -text4*.h5 filter=lfs diff=lfs merge=lfs -text5*.tflite filter=lfs diff=lfs merge=lfs -text6*.tar.gz filter=lfs diff=lfs merge=lfs -text7*.ot filter=lfs diff=lfs merge=lfs -text8*.onnx filter=lfs diff=lfs merge=lfs -text9*.arrow filter=lfs diff=lfs merge=lfs -text10*.ftz filter=lfs diff=lfs merge=lfs -text11*.joblib filter=lfs diff=lfs merge=lfs -text12*.model filter=lfs diff=lfs merge=lfs -text13*.msgpack filter=lfs diff=lfs merge=lfs -text14*.pb filter=lfs diff=lfs merge=lfs -text15*.pt filter=lfs diff=lfs merge=lfs -text16*.pth filter=lfs diff=lfs merge=lfs -text17