olmer/wiki_mpnet_index

Semantic search over the 44 million of English Wikipedia paragraphs using sentence transformers encoder. The dataset contains: 43 911 155 paragraphs from 6 458 670 wikipedia articles stored in a zip archive; FAISS index with the embeddings; Retriever module for semantic search over the paragraphs. The size of each paragraph varies from 20 to 2000 characters.The embedding vector size is 768.The index is 4-bit-quantized 2-level IVF16384_HNSW32 constructed with the FAISS… See the full description on the dataset page: https://huggingface.co/datasets/olmer/wiki_mpnet_index.