datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-edu
FineWeb-Edu (Lance Format)
A Lance-formatted version of FineWeb-Edu — over 1.5 billion educational web passages with cleaned text, source metadata, language detection signals, and 384-dim text embeddings — available directly from the Hub at hf://datasets/lance-format/fineweb-edu/data/train.lance.
Key features
Cleaned passage text in the text column with the source url and title carried alongside.
Language detection signals (language, language_probability) for filtered… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/fineweb-edu.mini-fineweb
Fineweb subset
[Work-In-Progress] Filtering by how a knowledge could be useful for pretraining small model
I'm targeting only 3-4% of original fineweb in size.