Datasetpublic

WebOrganizer/Corpus-200B

WebOrganizer/Corpus-200B [Paper] [Website] [GitHub] This dataset is a pre-processed version of the 1b-1x CommonCrawl pool from DataComps-LM cleaned with (1) RefinedWeb filters and (2) BFF deduplication. We provide the resulting 200B token corpus annotated with two quality scores, WebOrganizer domains, and k-means scores. Download the dataset by cloning the repository with Git LFS instead of HuggingFace's load_dataset(). The dataset has the following folder structure:… See the full description on the dataset page: https://huggingface.co/datasets/WebOrganizer/Corpus-200B.

sourceHugging Faceupdated 3mo agoView on Hugging Face

12likes39kdownloads

Dataset Card

WebOrganizer/Corpus-200B

[Paper] [Website] [GitHub]

This dataset is a pre-processed version of the 1b-1x CommonCrawl pool from DataComps-LM cleaned with (1) RefinedWeb filters and (2) BFF deduplication.

We provide the resulting 200B token corpus annotated with two quality scores, WebOrganizer domains, and k-means scores.

_Download the dataset by cloning the repository with Git LFS instead of HuggingFace's `loaddataset()`.__

The dataset has the following folder structure:

bash

Corpus-200B/
    documents/  # Pre-processed web documents
        - CC_shard_00000000_processed.jsonl.zst
        - CC_shard_00000001_processed.jsonl.zst
        - ...
    tokens/  # number of tokens per document (GPT-NeoX tokenizer)
        - CC_shard_00000000_processed.npy
        - CC_shard_00000001_processed.npy
        - ...
    scores_dclm-fasttext/  # DCLM-fasttext score
        - CC_shard_00000000_processed.npy
        - ...
    scores_fineweb-edu/  # FineWeb-Edu score
        - CC_shard_00000000_processed.npy
        - ...
    scores_fineweb-edu__rounded/  # Rounded FineWeb-Edu score
        - CC_shard_00000000_processed__rounded.npy
        - ...
    domains_topics/  # TopicClassifier annotations
        - CC_shard_00000000_processed__choice.npy  # index of top choice
        - ...
    domain_topics__logits/
        - CC_shard_00000000_processed__logits.npy  # logits for each topic
        - ...
    domains_formats/  # FormatClassifier annotations
        - CC_shard_00000000_processed__choice.npy  # index of top choice
        - ...
      domains_formats/  # FormatClassifier annotations
        - CC_shard_00000000_processed__logits.npy  # logits for each format
        - ...
    domains_clusters-k24/  # K-means clusters
        - CC_shard_00000000_processed.npy  # cluster assignment for each document
        - ...

We also include statistics about the presence and co-occurence of domains in the domain_statistics/ folder, computed with the domain_statistics.py script.

Citation

If you make use of this pre-processed corpus in your work, please cite:

bibtex

@article{wettig2025organize,
  title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},
  author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},
  journal={arXiv preprint arXiv:2502.10341},
  year={2025}
}