nleroy917/fineweb-bge-large-en-v1.5

Work in progress. Coverage is partial — additional CommonCrawl dumps are being embedded and uploaded incrementally.

FineWeb — BGE-Large-EN-v1.5 + BM25 (Pre-embedded)

Pre-computed dense and sparse embeddings for the FineWeb web corpus, ready for direct ingestion into a vector database (e.g. Qdrant).

Source dataset

FineWeb is a 15-trillion-token English web dataset derived from 96 CommonCrawl snapshots spanning Summer 2013 through June 2025. It was produced by Hugging Face using aggressive quality filtering (URL dedup, MinHash near-dedup, language filtering, custom heuristic filters, and a classifier distilled from Llama-3-70B-Instruct) to remove boilerplate, spam, and low-quality text while preserving high-quality prose.

Embeddings

Text is truncated to 512 tokens before dense encoding. Input text is capped at 8192 characters.

Schema

Each parquet row / Qdrant payload contains:

Coverage

Usage

These parquets are designed to be loaded directly into Qdrant with matching dense + sparse named vectors:

from qdrant_client import QdrantClient, models

client = QdrantClient(url="...", api_key="...")
# Collection expects:
#   vectors_config={"dense": VectorParams(size=1024, distance=Distance.COSINE)}
#   sparse_vectors_config={"sparse": SparseVectorParams()}

See vectorforge for the full embedding and loading pipeline.

Citation

If you use this dataset, please also cite the original FineWeb dataset:

@software{penedo2024fineweb,
  author = {Penedo, Guilherme and Kydlíček, Hynek and allal, Loubna Ben and Lozhkov, Anton and Mitchell, Margaret and Raffel, Colin and Von Werra, Leandro and Wolf, Thomas},
  title = {The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale},
  year = {2024},
  url = {https://huggingface.co/datasets/HuggingFaceFW/fineweb},
}