Datasetpublic

olmer/wiki_mpnet_index

Semantic search over the 44 million of English Wikipedia paragraphs using sentence transformers encoder. The dataset contains: 43 911 155 paragraphs from 6 458 670 wikipedia articles stored in a zip archive; FAISS index with the embeddings; Retriever module for semantic search over the paragraphs. The size of each paragraph varies from 20 to 2000 characters.The embedding vector size is 768.The index is 4-bit-quantized 2-level IVF16384_HNSW32 constructed with the FAISS… See the full description on the dataset page: https://huggingface.co/datasets/olmer/wiki_mpnet_index.

sourceHugging Facecc-by-sa-3.0updated 3y agoView on Hugging Face

1likes20downloads

Dataset Card

Semantic search over the 44 million of English Wikipedia paragraphs using sentence transformers encoder.

The dataset contains:

—43 911 155 paragraphs from 6 458 670 wikipedia articles stored in a zip archive;
—FAISS index with the embeddings;
—Retriever module for semantic search over the paragraphs.

The size of each paragraph varies from 20 to 2000 characters. The embedding vector size is 768. The index is 4-bit-quantized 2-level IVF16384_HNSW32 constructed with the FAISS library. Sentence encoder: all-mpnet-base-v2.