fineweb2
beetle-bilingual-balanced-b1-fineweb-spa-engbeetle-bilingual-l2-50-classroom-20-b4-fineweb-spa-engbeetle-bilingual-l2-50-sequential-33-67-b3-fineweb-spa-engbeetle-bilingual-l2-80-late-b5-fineweb-spa-engbeetle-bilingual-l2-50-sequential-33-67-b3-fineweb-zho-engbeetle-bilingual-l2-50-simultaneous-b2-fineweb-spa-engbeetle-bilingual-l2-50-simultaneous-b2-fineweb-zho-engbeetle-bilingual-l2-50-simultaneous-b2-fineweb-hin-eng
Datasets
All datasets matching “fineweb2”fineweb-2
🥂 FineWeb2
A sparkling update with 1000s of languages
What is it?
This is the second iteration of the popular 🍷 FineWeb dataset, bringing high quality pretraining data to over 1000 🗣️ languages.
The 🥂 FineWeb2 dataset is fully reproducible, available under the permissive ODC-By 1.0 license and extensively validated through hundreds of ablation experiments.
In particular, on the set of 9 diverse languages we used to guide our processing decisions, 🥂… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-2.fineweb-2-duckdbs
DuckDB datasets for (dump, id) querying on FineWeb 2
This repo contains some DuckDB databases to check whether a given WARC UID exists in a FineWeb-2 dump. Usage example
is given below, but note especially that if you are using URNs (likely, if you are working with CommonCrawl data),
then you first have to extract the UID (the id column is of type UUID in the databases).
Download
All files:
huggingface-cli download BramVanroy/fineweb-2-duckdbs --local-dir… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/fineweb-2-duckdbs.mmBERT-pretrain-p2-fineweb2-remaining
mmBERT Pre-training Data P2
Phase 1 of 3: Diverse multilingual pre-training data mixture (trained for 2.3T tokens) used to train the mmBERT model suite.
NOTE: this is only P2 of the pre-training data due to HF limits, you need to download and combine all three into one folderThis dataset contains the pre-training phase data used to train all mmBERT encoder models. The data is provided in MDS format ready for use with Composer and the ModernBERT training repository.… See the full description on the dataset page: https://huggingface.co/datasets/orionweller/mmBERT-pretrain-p2-fineweb2-remaining.FineWeb2-HQ
FineWeb2-HQ
Dataset summary
FineWeb2-HQ is a high-quality, model-filtered pretraining dataset derived as a subset of FineWeb2, spanning 20 languages. It enables around 6x faster pretraining compared to the base dataset. FineWeb2-HQ was created by selecting the top 10% quality documents of FineWeb2 in each language, based on scores assigned by a deep learning classifier trained to identify structured and knowledge-rich samples using XLM-RoBERTa embeddings.
Validation… See the full description on the dataset page: https://huggingface.co/datasets/epfml/FineWeb2-HQ.FineWeb2-embedded
FineWeb2-embedded
Dataset summary
FineWeb2-embedded is an extension of the FineWeb2 dataset, annotated with document-level XLM-RoBERTa embeddings for 20 languages, making the dataset useful for a variety of tasks, including document clustering, filtering, and other multilingual research.
Since XLM-RoBERTa has a sequence length limit of 512 tokens, each document's embeddings are obtained by mean-pooling 512 token chunks of the XLM-RoBERTa output. Therefore, longer texts… See the full description on the dataset page: https://huggingface.co/datasets/epfml/FineWeb2-embedded.FineWeb2-mds-tokenized-v2