WebOrganizer/FormatAnnotations-Llama-3.1-8B

WebOrganizer/FormatAnnotations-Llama-3.1-8B

[Paper] [Website] [GitHub]

This dataset contains 1M web pages annotated with format/type labels by the Llama-3.1-8B model. The web pages are a sample of the DCLM RefinedWeb reproduction. It is used as first-stage training data for the WebOrganizer/FormatClassifier.

Dataset Structure

Each example contains the following fields:

• —text: The text content of the web page
• —url: The URL of the web page
• —top_choice_index: Index of the most likely format label
• —top_choice: The most likely format label (using the description in the taxonomy config)
• —top_choice_prob: Probability of the most likely format
• —choice_probs: Probabilities for all possible format choices
• —choice_loss: Average log-likelihood of the answers
• —metadata: Additional metadata from the source dataset
• —label_permutation: Permutation used for the label ordering during prompting
• —fewshot_permutation: Permutation used for few-shot demonstration examples during prompting

Citation

@article{wettig2025organize,
  title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},
  author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},
  journal={arXiv preprint arXiv:2502.10341},
  year={2025}
}