r76941156/rare-disease-embedding-model

License

This model is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. It may be used, shared, and adapted for non-commercial research purposes only.

Qwen3-Based Rare Disease Biomedical Embedding Model

This repository provides a fine-tuned embedding model designed for biomedical and rare disease text representation. The repository provides two variants based on Qwen3-Embedding-8B, each fine-tuned using 1,320 disease summaries generated by either Claude Sonnet 4 or OpenAI o3 from the National Organization for Rare Disorders (NORD) disease catalog. The source code for model training and downstream ranking is available in the GEN-KnowRD GitHub repository.

We provide the Claude Sonnet 4 and OpenAI o3 knowledge fine-tuned versions in this repository. Researchers are welcome to further fine-tune or reproduce the training workflow using disease summaries generated by other LLMs or alternative rare disease knowledge sources.

Intended use

This model is intended for research use in rare disease phenotype representation, disease-profile retrieval, and embedding-based candidate ranking.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("r76941156/rare-disease-embedding-model")

# To use the OpenAI o3 knowledge fine-tuned variant instead:
# model = SentenceTransformer(
#     "r76941156/rare-disease-embedding-model",
#     revision="o3"
# )

texts = [
    "progressive dyspnea and pulmonary fibrosis",
    "idiopathic pulmonary fibrosis with cough and exertional shortness of breath",
    "seizures, developmental delay, and skin lesions suggestive of tuberous sclerosis",
    "recurrent infections and low immunoglobulin levels"
]

embeddings = model.encode(texts, normalize_embeddings=True)

similarities = model.similarity(embeddings, embeddings)
print("Embedding shape:", embeddings.shape)
print("Similarity matrix shape:", similarities.shape)
print(similarities)

Performance

Compared with the original Qwen3-Embedding-8B model, the fine-tuned model improved Recall@1 by 0.59 to 6.65 percentage points on the combined benchmark, depending on the disease knowledge source. Performance varied across benchmark subsets.

Citation

Yan C, Su WC, Xin Y, Grabowska ME, Kerchberger VE, Borza VA, Wang J, Wang L, Li R, Lynn J, Dickson AL. Reframing AI for Rare Disease Recognition. Research Square. 2026 Apr 2:rs-3.