TurkuNLP/WebDocumentDescriptors

[image]

Data release for the paper Task-Agnostic Web Document Annotation with LLM-Generated Descriptors (forthcoming).

The descriptors are generated via a task-agnostic data annotation pipeline described in the paper (link coming soon).

This Hugging Face dataset repository contains 5 distinct datasets:

1. 1.a descriptor-annotated version of a 10 billion token (~15 million document) sample of FineWeb.
2. 2.The 800k label descriptor schema
3. 3.The 500k document sample of FineWeb used to develop the schema along with the descriptor annotations and metadata from the generation.
4. 4.The 100k document sample of FineWeb used to evaluate the schema along with the descriptor annotations and metadata from the generation.
5. 5.Merge traces, i.e. the final schema with pointers to the original raw descriptors that lead to the final descriptor in the merging steps.

Citing

If you use this data please cite the original FineWeb paper and the our paper.

Task-Agnostic Web Document Annotation with LLM-Generated Descriptors

@inproceedings{
tarkkaetal-descriptors,
title={Task-Agnostic Web Document Annotation with LLM-Generated Descriptors},
author={Tarkka, Otto and Henriksson, Erik and Kanerva, Jenna and Ginter, Filip},
year={2026 (forthcoming)},
}

FineWeb: Link to ArXiv: https://arxiv.org/abs/2406.17557

@inproceedings{
penedo2024the,
title={The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale},
author={Guilherme Penedo and Hynek Kydl{\'\i}{\v{c}}ek and Loubna Ben allal and Anton Lozhkov and Margaret Mitchell and Colin Raffel and Leandro Von Werra and Thomas Wolf},
booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2024},
url={https://openreview.net/forum?id=n6SCkn2QaG}
}