CoolFace
Datasetpublic

aaekay/pubmed-embeddings

PubMed Embedding Vectors This dataset contains embedding vectors generated from local PubMed title and abstract text. It is designed for biomedical retrieval and nearest-neighbor research. The public files intentionally do not include PubMed titles, abstracts, or full text. Rows contain PMIDs, embeddings, hashes, and lightweight metadata so researchers can join against their own authorized PubMed mirror or the official NCBI/PubMed services. Configs Config… See the full description on the dataset page: https://huggingface.co/datasets/aaekay/pubmed-embeddings.

sourceHugging Facecc-by-4.0updated 4mo agoView on Hugging Face
0likes280downloads
Dataset Card

PubMed Embedding Vectors

This dataset contains embedding vectors generated from local PubMed title and abstract text. It is designed for biomedical retrieval and nearest-neighbor research.

The public files intentionally do not include PubMed titles, abstracts, or full text. Rows contain PMIDs, embeddings, hashes, and lightweight metadata so researchers can join against their own authorized PubMed mirror or the official NCBI/PubMed services.

Configs

ConfigModelDimRowsQdrant collection
neumlpubmedbertbase_embeddingsNeuML/pubmedbert-base-embeddings76828,460,827pubmed_emb_neuml_pubmedbert_base_embeddings_ddbc790c
qwen3embedding0_6bQwen/Qwen3-Embedding-0.6B102428,460,827pubmed_emb_qwen_qwen3_embedding_0_6b_cdca07b0

Columns

  • —pmid: PubMed identifier.
  • —embedding: fixed-size embedding vector for the selected config.
  • —text_sha256: SHA-256 of the local title+abstract text used for embedding, when available.
  • —pub_year, pub_month: publication date metadata, when available.
  • —raw_token_count, used_token_count, was_truncated: embedding input token metadata, when available.

Usage

python
from datasets import load_dataset

repo_id = "aaekay/pubmed-embedding"
ds = load_dataset(repo_id, "qwen3_embedding_0_6b", split="train", streaming=True)
row = next(iter(ds))
print(row["pmid"], len(row["embedding"]))

Use the PMID to retrieve citation details from PubMed:

python
pmid = row["pmid"]
url = f"https://pubmed.ncbi.nlm.nih.gov/{pmid}/"

Source And Redistribution Notes

  • —Source records come from a local PubMed baseline/update mirror.
  • —NLM notes that PubMed abstracts may be protected by third-party copyright, so this dataset excludes article titles, abstracts, and full text.
  • —The generated embedding dataset is released as cc-by-4.0; upstream PubMed records and embedding models remain subject to their own terms.

Relevant upstream documentation:

  • —PubMed download page: https://pubmed.ncbi.nlm.nih.gov/download/
  • —NLM copyright information: https://www.nlm.nih.gov/databases/download.html
  • —Hugging Face large repository guidance: https://huggingface.co/docs/hub/storage-limits

Manifest

Export metadata, shard checksums, and source collection details are stored in metadata/manifest.json. The public schema is stored in metadata/schema.json.

aaekay/pubmed-embeddings · CoolFace