Datasetpublic

olmer/wiki_mpnet_embeddings

Embeddings of the english Wikipedia paragraphs using all-mpnet-base-v2 sentence transformers encoder.The dataset contains 43 911 155 paragraphs from 6 458 670 Wikipedia articles.The size of each paragraph varies from 20 to 2000 characters.For each paragraph there is an embedding of size 768.Embeddings are stored in numpy files, 1 000 000 embeddings per file.For each embedding file, there is an ids file that contains the list of ids of the corresponding paragraphs.Be careful, dataset size is… See the full description on the dataset page: https://huggingface.co/datasets/olmer/wiki_mpnet_embeddings.

sourceHugging Facecc-by-sa-3.0updated 3y agoView on Hugging Face

1likes99downloads

Dataset Card

Embeddings of the english Wikipedia paragraphs using all-mpnet-base-v2 sentence transformers encoder. The dataset contains 43 911 155 paragraphs from 6 458 670 Wikipedia articles. The size of each paragraph varies from 20 to 2000 characters. For each paragraph there is an embedding of size 768. Embeddings are stored in numpy files, 1 000 000 embeddings per file. For each embedding file, there is an ids file that contains the list of ids of the corresponding paragraphs. _Be careful, dataset size is 151Gb_.