Datasetpublic

WebOrganizer/Corpus-200B

WebOrganizer/Corpus-200B [Paper] [Website] [GitHub] This dataset is a pre-processed version of the 1b-1x CommonCrawl pool from DataComps-LM cleaned with (1) RefinedWeb filters and (2) BFF deduplication. We provide the resulting 200B token corpus annotated with two quality scores, WebOrganizer domains, and k-means scores. Download the dataset by cloning the repository with Git LFS instead of HuggingFace's load_dataset(). The dataset has the following folder structure:… See the full description on the dataset page: https://huggingface.co/datasets/WebOrganizer/Corpus-200B.

sourceHugging Faceupdated 3mo agoView on Hugging Face

12likes34kdownloads

50 commits on main

78668463mo ago

add missing files

PrincetonPLI

c33d3943mo ago

add missing files

PrincetonPLI

9d8e90a3mo ago

upload missing score files

PrincetonPLI

25f7f562y ago

Update README.md

awettig

669caf12y ago

Update README.md

awettig

6722f7d2y ago

Add files using upload-large-folder tool

princeton-nlp

e5e8b7f2y ago

Add files using upload-large-folder tool

princeton-nlp

1640eae2y ago

Add files using upload-large-folder tool

princeton-nlp

a4b6cc62y ago

Add files using upload-large-folder tool

princeton-nlp

20d616a2y ago

Add files using upload-large-folder tool

princeton-nlp

778062b2y ago

Add files using upload-large-folder tool

princeton-nlp

40afd0b2y ago

Add files using upload-large-folder tool

princeton-nlp

a3211b02y ago

Add files using upload-large-folder tool

princeton-nlp

eb681862y ago

Add files using upload-large-folder tool

princeton-nlp

dacf1a62y ago

Add files using upload-large-folder tool

princeton-nlp

5dc31052y ago

Add files using upload-large-folder tool

princeton-nlp

a8ccb6d2y ago

Add files using upload-large-folder tool

princeton-nlp

0842dc82y ago

Add files using upload-large-folder tool

princeton-nlp

96dc4ce2y ago

Add files using upload-large-folder tool

princeton-nlp

92ccb172y ago

Add files using upload-large-folder tool

princeton-nlp

ca8d2dd2y ago

Add files using upload-large-folder tool

princeton-nlp

b4f4c442y ago

Add files using upload-large-folder tool

princeton-nlp

a4643882y ago

Add files using upload-large-folder tool

princeton-nlp

cd36b012y ago

Add files using upload-large-folder tool

princeton-nlp

2d209542y ago

Add files using upload-large-folder tool

princeton-nlp

d8239502y ago

Add files using upload-large-folder tool

princeton-nlp

1815aab2y ago

Add files using upload-large-folder tool

princeton-nlp

8a2ddc32y ago

Add files using upload-large-folder tool

princeton-nlp

b10cdb22y ago

Add files using upload-large-folder tool

princeton-nlp

170fffa2y ago

Add files using upload-large-folder tool

princeton-nlp

9a72ae82y ago

Add files using upload-large-folder tool

princeton-nlp

984d4ec2y ago

Add files using upload-large-folder tool

princeton-nlp

1a06ebc2y ago

Add files using upload-large-folder tool

princeton-nlp

499ad692y ago

Add files using upload-large-folder tool

princeton-nlp

59780e92y ago

Add files using upload-large-folder tool

princeton-nlp

7d595b22y ago

Add files using upload-large-folder tool

princeton-nlp

3a4520b2y ago

Add files using upload-large-folder tool

princeton-nlp

4e3e9df2y ago

Add files using upload-large-folder tool

princeton-nlp

5d4e9312y ago

Add files using upload-large-folder tool

princeton-nlp

ecd29f42y ago

Add files using upload-large-folder tool

princeton-nlp

8bd69742y ago

Add files using upload-large-folder tool

princeton-nlp

e85041e2y ago

Add files using upload-large-folder tool

princeton-nlp

fa202112y ago

Add files using upload-large-folder tool

princeton-nlp

add01072y ago

Add files using upload-large-folder tool

princeton-nlp

f84f0f32y ago

Add files using upload-large-folder tool

princeton-nlp

17e11272y ago

Add files using upload-large-folder tool

princeton-nlp

48675312y ago

Add files using upload-large-folder tool

princeton-nlp

935ba4f2y ago

Add files using upload-large-folder tool

princeton-nlp

55846fe2y ago

Add files using upload-large-folder tool

princeton-nlp

02241f92y ago

Add files using upload-large-folder tool

princeton-nlp

WebOrganizer/Corpus-200B · CoolFace